2024-03-24T10:22:34.917 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T10:22:34.918 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053 --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053/orig.config.yaml 2024-03-24T10:22:34.996 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:22:35.089 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053 branch: wip-batrick-testing-20240323.003144-squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7619053' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 21b3bd672163092b4e542f772797fcba02d3d4de ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXzipRrH1SrY8FYg9ynf3fYPhWZ9oKyl/rry103egfreIS99qJD5Kob2MXPsPUJxBAw7U47PlFSUEY97xPsn/A= smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIArAC655sYkrF00LrnFlueHTSqYgeStBPd1xrTiOorhQQoDRABWhPCf8swRSnVSvdCKUWgPoL0YCt621/Z73TM= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-24T10:22:35.090 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T10:22:35.090 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T10:22:35.090 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T10:22:35.092 INFO:teuthology.task.internal:Checking packages... 2024-03-24T10:22:35.114 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T10:22:35.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:22:35.114 INFO:teuthology.packaging:ref: None 2024-03-24T10:22:35.114 INFO:teuthology.packaging:tag: None 2024-03-24T10:22:35.114 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T10:22:35.114 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:22:35.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T10:22:35.327 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T10:22:35.329 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T10:22:35.335 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T10:22:35.335 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T10:22:35.340 INFO:teuthology.task.internal:Saving configuration 2024-03-24T10:22:35.361 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T10:22:35.366 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:22:35.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:19.204852', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXzipRrH1SrY8FYg9ynf3fYPhWZ9oKyl/rry103egfreIS99qJD5Kob2MXPsPUJxBAw7U47PlFSUEY97xPsn/A='} 2024-03-24T10:22:35.415 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:19.206375', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIArAC655sYkrF00LrnFlueHTSqYgeStBPd1xrTiOorhQQoDRABWhPCf8swRSnVSvdCKUWgPoL0YCt621/Z73TM='} 2024-03-24T10:22:35.416 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T10:22:35.421 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-24T10:22:35.421 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-24T10:22:35.421 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T10:22:35.489 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2c7a5a2160>, signals=[15]) 2024-03-24T10:22:35.489 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T10:22:35.495 INFO:teuthology.task.internal:Opening connections... 2024-03-24T10:22:35.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:22:35.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:35.601 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-03-24T10:22:35.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:35.708 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T10:22:35.715 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-24T10:22:35.733 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-24T10:22:35.733 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:22:35.791 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:22:35.792 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:22:35.792 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-24T10:22:35.817 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-03-24T10:22:35.836 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-03-24T10:22:35.836 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:22:35.895 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:22:35.896 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:22:35.896 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:22:35.896 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-03-24T10:22:35.925 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T10:22:35.935 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T10:22:35.940 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T10:22:35.941 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:22:35.943 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:22:35.960 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T10:22:35.998 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T10:22:35.998 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:22:36.001 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:22:36.030 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T10:22:36.086 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T10:22:36.100 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T10:22:36.101 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T10:22:36.101 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:36.101 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:36.118 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-24T10:22:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:36.119 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-24T10:22:36.175 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:22:36.175 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-361.el9.x86_64 2024-03-24T10:22:36.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-03-24T10:22:36.177 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:22:36.177 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2024-03-24T10:22:36.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-03-24T10:22:38.202 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T10:22:38.375 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T10:22:39.711 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-24T10:22:39.808 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-24T10:22:41.549 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-24T10:22:43.502 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T10:22:45.126 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-24T10:22:54.791 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 814 kB/s | 6.1 MB 00:07 2024-03-24T10:22:56.275 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-24T10:23:02.458 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-24T10:23:06.733 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-24T10:23:08.385 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-24T10:23:09.469 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-24T10:23:12.994 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 21 MB 00:16 2024-03-24T10:23:14.661 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-24T10:23:16.859 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:23:16.859 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:23:16.953 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:23:16.963 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-03-24T10:23:16.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:23:16.964 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 76 M 2024-03-24T10:23:16.964 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 123 M 2024-03-24T10:23:16.964 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:23:17.179 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 294 kB/s | 44 kB 00:00 2024-03-24T10:23:17.762 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-24T10:23:18.099 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T10:23:18.263 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 25 MB/s | 27 MB 00:01 2024-03-24T10:23:18.571 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 22 MB/s | 33 MB 00:01 2024-03-24T10:23:18.571 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:23:18.571 INFO:teuthology.orchestra.run.smithi026.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-24T10:23:19.267 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:23:19.671 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:23:19.671 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:23:20.306 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:23:20.306 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:23:20.412 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:23:20.422 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 76 M 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 123 M 2024-03-24T10:23:20.423 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:23:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-24T10:23:20.742 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:23:20.742 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:23:22.517 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.5 MB/s | 15 MB 00:01 2024-03-24T10:23:23.354 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:23:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:23:24.087 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:24.133 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 9.3 MB/s | 33 MB 00:03 2024-03-24T10:23:24.775 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.7 MB/s | 27 MB 00:04 2024-03-24T10:23:24.776 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:23:24.776 INFO:teuthology.orchestra.run.smithi031.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-24T10:23:25.473 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:23:25.930 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:23:25.931 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:23:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:26.972 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:23:26.972 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:23:27.089 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:29.594 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:23:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:23:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:32.159 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:32.206 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:23:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:33.351 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:23:38.451 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:38.493 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:23:43.628 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:08.302 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:08.302 INFO:teuthology.orchestra.run.smithi026.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:24:08.302 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:24:08.352 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:24:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:24:09.103 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:24:09.404 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:09.552 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:09.553 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:24:09.553 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:24:09.553 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:24:09.553 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-24T10:24:09.553 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:24:09.553 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:09.635 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:09.636 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:09.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-03-24T10:24:09.677 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:24:09.678 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-03-24T10:24:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:24:09.679 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:24:09.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:24:09.679 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:24:09.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:24:10.328 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-03-24T10:24:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:24:11.765 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-03-24T10:24:11.768 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-24T10:24:11.783 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.bARFkaXXos 2024-03-24T10:24:11.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bARFkaXXos 2024-03-24T10:24:11.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.bARFkaXXos 2024-03-24T10:24:11.995 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.bARFkaXXos is 6KB 2024-03-24T10:24:12.044 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.bARFkaXXos 2024-03-24T10:24:12.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:24:12.127 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:24:12.127 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:24:12.127 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:24:12.127 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:24:12.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:24:12.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-03-24T10:24:14.375 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:14.376 INFO:teuthology.orchestra.run.smithi031.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:24:14.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:24:14.422 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:14.803 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:14.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:24:14.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:14.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:15.219 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.221 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:24:15.221 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:24:15.614 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:15.731 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.732 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:15.732 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:24:15.732 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:24:15.732 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:24:15.732 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-24T10:24:15.732 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:24:15.732 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:15.815 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:15.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:15.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:24:15.855 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-03-24T10:24:15.856 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:24:15.857 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:24:15.859 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:24:15.859 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:24:15.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:24:16.461 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-03-24T10:24:17.886 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:24:17.907 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-03-24T10:24:17.909 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-03-24T10:24:17.924 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.ywyE5d1RIV 2024-03-24T10:24:17.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ywyE5d1RIV 2024-03-24T10:24:17.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.ywyE5d1RIV 2024-03-24T10:24:18.144 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.ywyE5d1RIV is 6KB 2024-03-24T10:24:18.192 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.ywyE5d1RIV 2024-03-24T10:24:18.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:24:18.275 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:24:18.275 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:24:18.275 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:24:18.276 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:24:18.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:24:18.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-03-24T10:24:42.275 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-24T10:24:42.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-24T10:24:42.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:48.437 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-24T10:24:48.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T10:24:48.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:42.330 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:25:48.449 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:25:51.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-24T10:25:51.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:51.704 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-24T10:25:52.096 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-24T10:25:52.096 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:25:52.096 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-24T10:25:52.111 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:25:52.112 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:25:52.112 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:25:52.112 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:25:53.113 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-24T10:25:53.113 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:25:53.167 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-03-24T10:25:53.210 DEBUG:teuthology.parallel:result is None 2024-03-24T10:25:57.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T10:25:57.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:57.748 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-24T10:25:58.129 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T10:25:58.129 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:25:58.129 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-24T10:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:25:58.145 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:25:58.145 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:25:58.145 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:25:59.145 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-24T10:25:59.145 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:25:59.202 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-03-24T10:25:59.237 DEBUG:teuthology.parallel:result is None 2024-03-24T10:25:59.238 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T10:25:59.247 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T10:25:59.247 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:25:59.250 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:25:59.267 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T10:25:59.274 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T10:25:59.280 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T10:25:59.281 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:25:59.308 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:25:59.352 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T10:25:59.359 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T10:25:59.359 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:25:59.387 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:25:59.420 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:25:59.434 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:25:59.437 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:25:59.438 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T10:25:59.446 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T10:25:59.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:25:59.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:25:59.508 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T10:25:59.517 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T10:25:59.517 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:25:59.548 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:25:59.565 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:25:59.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:25:59.691 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:25:59.775 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:25:59.816 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:25:59.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:25:59.884 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:25:59.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:25:59.979 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:00.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:00.107 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:26:00.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:26:00.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-03-24T10:26:00.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-03-24T10:26:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:26:00.266 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:26:00.742 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T10:26:00.750 INFO:teuthology.task.internal:Starting timer... 2024-03-24T10:26:00.751 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T10:26:00.760 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T10:26:00.768 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T10:26:00.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-03-24T10:26:00.845 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-03-24T10:26:00.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-03-24T10:26:00.908 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-03-24T10:26:00.910 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T10:26:00.910 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-03-24T10:26:00.948 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-03-24T10:26:00.948 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-03-24T10:26:00.973 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-03-24T10:26:00.974 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2024-03-24T10:26:00.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:26:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:26:01.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:26:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:26:01.047 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T10:26:01.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:26:01.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:26:01.125 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T10:26:01.135 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T10:26:01.151 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T10:26:01.243 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T10:30:20.622 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2024-03-24T10:30:20.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-24T10:30:20.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:20.722 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-24T10:30:20.805 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-24T10:30:20.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T10:30:20.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:20.903 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-24T10:30:20.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T10:30:20.988 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T10:30:21.002 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T10:30:21.002 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:30:21.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:30:21.005 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:30:21.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:30:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:30:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:30:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:30:21.079 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:30:21.080 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-03-24T10:30:21.094 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-03-24T10:30:21.109 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:30:21.110 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-03-24T10:30:21.124 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-03-24T10:30:21.125 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:30:21.141 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:30:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:30:21.178 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.182 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.212 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:30:21.215 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:30:21.215 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-24T10:30:21.215 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.216 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.216 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.216 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:21.217 INFO:teuthology.run_tasks:Running task install... 2024-03-24T10:30:21.230 DEBUG:teuthology.task.install:project ceph 2024-03-24T10:30:21.230 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T10:30:21.230 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T10:30:21.230 INFO:teuthology.task.install:Using flavor: default 2024-03-24T10:30:21.245 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T10:30:21.246 INFO:teuthology.task.install:extra packages: [] 2024-03-24T10:30:21.257 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:30:21.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:30:21.258 INFO:teuthology.packaging:ref: None 2024-03-24T10:30:21.258 INFO:teuthology.packaging:tag: None 2024-03-24T10:30:21.258 INFO:teuthology.packaging:branch: reef 2024-03-24T10:30:21.258 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:21.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T10:30:21.260 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:30:21.260 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:30:21.260 INFO:teuthology.packaging:ref: None 2024-03-24T10:30:21.260 INFO:teuthology.packaging:tag: None 2024-03-24T10:30:21.260 INFO:teuthology.packaging:branch: reef 2024-03-24T10:30:21.260 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:21.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T10:30:21.541 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/ 2024-03-24T10:30:21.542 INFO:teuthology.task.install.rpm:Package version is 18.2.2-952.g738c2fd6 2024-03-24T10:30:21.614 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/ 2024-03-24T10:30:21.615 INFO:teuthology.task.install.rpm:Package version is 18.2.2-952.g738c2fd6 2024-03-24T10:30:21.733 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:30:21.733 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:30:21.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:30:21.738 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:30:21.738 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:30:21.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:30:21.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T10:30:21.765 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:30:21.765 INFO:teuthology.packaging:ref: None 2024-03-24T10:30:21.765 INFO:teuthology.packaging:tag: None 2024-03-24T10:30:21.765 INFO:teuthology.packaging:branch: reef 2024-03-24T10:30:21.765 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:21.766 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:30:21.767 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T10:30:21.767 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:30:21.767 INFO:teuthology.packaging:ref: None 2024-03-24T10:30:21.767 INFO:teuthology.packaging:tag: None 2024-03-24T10:30:21.767 INFO:teuthology.packaging:branch: reef 2024-03-24T10:30:21.767 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:21.767 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:30:21.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:30:21.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:30:21.919 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:30:21.920 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:30:21.987 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-03-24T10:30:21.987 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-03-24T10:30:21.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-03-24T10:30:21.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-03-24T10:30:22.235 INFO:teuthology.orchestra.run.smithi026.stdout:74 files removed 2024-03-24T10:30:22.236 INFO:teuthology.orchestra.run.smithi031.stdout:74 files removed 2024-03-24T10:30:22.260 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-03-24T10:30:22.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-03-24T10:30:22.975 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 190 kB/s | 77 kB 00:00 2024-03-24T10:30:23.024 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 168 kB/s | 77 kB 00:00 2024-03-24T10:30:23.400 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 30 kB/s | 9.9 kB 00:00 2024-03-24T10:30:23.435 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 24 kB/s | 9.9 kB 00:00 2024-03-24T10:30:23.817 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-24T10:30:23.960 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-24T10:30:24.244 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T10:30:24.362 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T10:30:25.778 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-24T10:30:25.821 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-24T10:30:26.619 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-24T10:30:27.295 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-03-24T10:30:29.013 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-24T10:30:29.814 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-24T10:30:34.341 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-24T10:30:35.295 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.1 MB 00:02 2024-03-24T10:30:35.814 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-24T10:30:36.779 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-24T10:30:37.135 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-24T10:30:42.266 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-24T10:30:44.754 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:30:44.773 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:30:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:30:44.773 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:30:44.773 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 7.7 M 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.3 M 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 5.1 M 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 18 M 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 25 k 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-24T10:30:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 702 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 474 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.5 M 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:30:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 45 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 161 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 321 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 298 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 99 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:30:44.776 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:30:44.777 INFO:teuthology.orchestra.run.smithi031.stdout:Install 28 Packages 2024-03-24T10:30:44.777 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-03-24T10:30:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:30:44.778 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 53 M 2024-03-24T10:30:44.778 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:(1/30): ceph-base-18.2.2-952.g738c2fd6.el9.x86_ 5.3 MB/s | 5.1 MB 00:00 2024-03-24T10:30:45.942 INFO:teuthology.orchestra.run.smithi031.stdout:(2/30): ceph-selinux-18.2.2-952.g738c2fd6.el9.x 432 kB/s | 25 kB 00:00 2024-03-24T10:30:46.076 INFO:teuthology.orchestra.run.smithi031.stdout:(3/30): ceph-radosgw-18.2.2-952.g738c2fd6.el9.x 6.7 MB/s | 7.7 MB 00:01 2024-03-24T10:30:46.109 INFO:teuthology.orchestra.run.smithi031.stdout:(4/30): libcephfs2-18.2.2-952.g738c2fd6.el9.x86 4.1 MB/s | 702 kB 00:00 2024-03-24T10:30:46.151 INFO:teuthology.orchestra.run.smithi031.stdout:(5/30): libradosstriper1-18.2.2-952.g738c2fd6.e 6.2 MB/s | 474 kB 00:00 2024-03-24T10:30:46.219 INFO:teuthology.orchestra.run.smithi031.stdout:(6/30): python3-ceph-argparse-18.2.2-952.g738c2 674 kB/s | 45 kB 00:00 2024-03-24T10:30:46.319 INFO:teuthology.orchestra.run.smithi031.stdout:(7/30): python3-ceph-common-18.2.2-952.g738c2fd 1.2 MB/s | 127 kB 00:00 2024-03-24T10:30:46.411 INFO:teuthology.orchestra.run.smithi031.stdout:(8/30): python3-cephfs-18.2.2-952.g738c2fd6.el9 1.7 MB/s | 161 kB 00:00 2024-03-24T10:30:46.519 INFO:teuthology.orchestra.run.smithi031.stdout:(9/30): python3-rados-18.2.2-952.g738c2fd6.el9. 2.9 MB/s | 321 kB 00:00 2024-03-24T10:30:46.725 INFO:teuthology.orchestra.run.smithi031.stdout:(10/30): ceph-common-18.2.2-952.g738c2fd6.el9.x 10 MB/s | 18 MB 00:01 2024-03-24T10:30:46.763 INFO:teuthology.orchestra.run.smithi031.stdout:(11/30): python3-rbd-18.2.2-952.g738c2fd6.el9.x 1.2 MB/s | 298 kB 00:00 2024-03-24T10:30:46.787 INFO:teuthology.orchestra.run.smithi031.stdout:(12/30): python3-rgw-18.2.2-952.g738c2fd6.el9.x 1.5 MB/s | 99 kB 00:00 2024-03-24T10:30:46.845 INFO:teuthology.orchestra.run.smithi031.stdout:(13/30): librgw2-18.2.2-952.g738c2fd6.el9.x86_6 6.1 MB/s | 4.5 MB 00:00 2024-03-24T10:30:46.971 INFO:teuthology.orchestra.run.smithi031.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-24T10:30:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 450 kB/s | 44 kB 00:00 2024-03-24T10:30:47.021 INFO:teuthology.orchestra.run.smithi031.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 679 kB/s | 159 kB 00:00 2024-03-24T10:30:47.121 INFO:teuthology.orchestra.run.smithi031.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-24T10:30:47.147 INFO:teuthology.orchestra.run.smithi031.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-24T10:30:47.172 INFO:teuthology.orchestra.run.smithi031.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 274 kB/s | 41 kB 00:00 2024-03-24T10:30:47.314 INFO:teuthology.orchestra.run.smithi031.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 579 kB/s | 111 kB 00:00 2024-03-24T10:30:47.339 INFO:teuthology.orchestra.run.smithi031.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-03-24T10:30:47.364 INFO:teuthology.orchestra.run.smithi031.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 517 kB/s | 26 kB 00:00 2024-03-24T10:30:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-03-24T10:30:47.432 INFO:teuthology.orchestra.run.smithi031.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-24T10:30:47.549 INFO:teuthology.orchestra.run.smithi031.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-24T10:30:47.724 INFO:teuthology.orchestra.run.smithi031.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-24T10:30:47.991 INFO:teuthology.orchestra.run.smithi031.stdout:(27/30): librados2-18.2.2-952.g738c2fd6.el9.x86 12 MB/s | 3.3 MB 00:00 2024-03-24T10:30:48.033 INFO:teuthology.orchestra.run.smithi031.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-03-24T10:30:48.192 INFO:teuthology.orchestra.run.smithi031.stdout:(29/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-03-24T10:30:48.250 INFO:teuthology.orchestra.run.smithi031.stdout:(30/30): librbd1-18.2.2-952.g738c2fd6.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-03-24T10:30:48.254 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:30:48.254 INFO:teuthology.orchestra.run.smithi031.stdout:Total 15 MB/s | 53 MB 00:03 2024-03-24T10:30:48.255 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:30:48.296 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:30:48.296 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:30:48.769 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:30:48.769 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:30:50.316 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:30:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-24T10:30:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-24T10:30:50.531 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-24T10:30:50.979 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-24T10:30:51.046 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:30:51.121 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:30:51.207 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T10:30:51.247 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:30:51.293 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:30:51.322 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-24T10:30:51.357 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 9/32 2024-03-24T10:30:51.426 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T10:30:51.458 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:30:51.516 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:30:51.625 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-24T10:30:51.657 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-24T10:30:51.697 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-24T10:30:51.730 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-24T10:30:51.757 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-24T10:30:51.793 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-24T10:30:51.826 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-24T10:30:52.149 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-24T10:30:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T10:30:52.246 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T10:30:52.277 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 21/32 2024-03-24T10:30:52.321 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-24T10:30:52.753 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-24T10:30:52.775 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T10:30:52.854 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T10:30:52.924 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 25/32 2024-03-24T10:30:52.952 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 26/32 2024-03-24T10:30:53.532 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 21 MB 00:16 2024-03-24T10:30:55.112 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:30:55.141 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:30:55.693 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:30:55.738 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T10:30:55.818 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T10:30:55.819 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:30:55.819 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:30:55.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:30:55.846 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:30:58.598 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T10:31:01.092 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 7.7 M 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.3 M 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:31:01.106 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 5.1 M 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 18 M 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 25 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 702 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:31:01.107 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 474 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.5 M 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 45 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 161 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 321 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 298 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 99 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:31:01.108 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:Install 28 Packages 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 53 M 2024-03-24T10:31:01.109 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:31:02.247 INFO:teuthology.orchestra.run.smithi026.stdout:(1/30): ceph-radosgw-18.2.2-952.g738c2fd6.el9.x 8.1 MB/s | 7.7 MB 00:00 2024-03-24T10:31:02.314 INFO:teuthology.orchestra.run.smithi026.stdout:(2/30): ceph-selinux-18.2.2-952.g738c2fd6.el9.x 377 kB/s | 25 kB 00:00 2024-03-24T10:31:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:(3/30): libcephfs2-18.2.2-952.g738c2fd6.el9.x86 9.1 MB/s | 702 kB 00:00 2024-03-24T10:31:02.489 INFO:teuthology.orchestra.run.smithi026.stdout:(4/30): libradosstriper1-18.2.2-952.g738c2fd6.e 4.6 MB/s | 474 kB 00:00 2024-03-24T10:31:02.789 INFO:teuthology.orchestra.run.smithi026.stdout:(5/30): librgw2-18.2.2-952.g738c2fd6.el9.x86_64 15 MB/s | 4.5 MB 00:00 2024-03-24T10:31:03.081 INFO:teuthology.orchestra.run.smithi026.stdout:(6/30): python3-ceph-argparse-18.2.2-952.g738c2 154 kB/s | 45 kB 00:00 2024-03-24T10:31:03.206 INFO:teuthology.orchestra.run.smithi026.stdout:(7/30): python3-ceph-common-18.2.2-952.g738c2fd 1.0 MB/s | 127 kB 00:00 2024-03-24T10:31:03.290 INFO:teuthology.orchestra.run.smithi026.stdout:(8/30): python3-cephfs-18.2.2-952.g738c2fd6.el9 1.9 MB/s | 161 kB 00:00 2024-03-24T10:31:03.357 INFO:teuthology.orchestra.run.smithi026.stdout:(9/30): ceph-base-18.2.2-952.g738c2fd6.el9.x86_ 2.5 MB/s | 5.1 MB 00:02 2024-03-24T10:31:03.383 INFO:teuthology.orchestra.run.smithi026.stdout:(10/30): python3-rados-18.2.2-952.g738c2fd6.el9 3.4 MB/s | 321 kB 00:00 2024-03-24T10:31:03.424 INFO:teuthology.orchestra.run.smithi026.stdout:(11/30): python3-rbd-18.2.2-952.g738c2fd6.el9.x 4.4 MB/s | 298 kB 00:00 2024-03-24T10:31:03.441 INFO:teuthology.orchestra.run.smithi026.stdout:(12/30): python3-rgw-18.2.2-952.g738c2fd6.el9.x 1.7 MB/s | 99 kB 00:00 2024-03-24T10:31:03.566 INFO:teuthology.orchestra.run.smithi026.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-24T10:31:03.592 INFO:teuthology.orchestra.run.smithi026.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-24T10:31:03.609 INFO:teuthology.orchestra.run.smithi026.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T10:31:03.684 INFO:teuthology.orchestra.run.smithi026.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-24T10:31:03.709 INFO:teuthology.orchestra.run.smithi026.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-24T10:31:03.726 INFO:teuthology.orchestra.run.smithi026.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-03-24T10:31:03.868 INFO:teuthology.orchestra.run.smithi026.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 699 kB/s | 111 kB 00:00 2024-03-24T10:31:03.910 INFO:teuthology.orchestra.run.smithi026.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-24T10:31:03.961 INFO:teuthology.orchestra.run.smithi026.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-03-24T10:31:04.011 INFO:teuthology.orchestra.run.smithi026.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-03-24T10:31:04.061 INFO:teuthology.orchestra.run.smithi026.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-24T10:31:04.370 INFO:teuthology.orchestra.run.smithi026.stdout:(24/30): ceph-common-18.2.2-952.g738c2fd6.el9.x 6.0 MB/s | 18 MB 00:03 2024-03-24T10:31:04.403 INFO:teuthology.orchestra.run.smithi026.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-24T10:31:04.537 INFO:teuthology.orchestra.run.smithi026.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-24T10:31:04.620 INFO:teuthology.orchestra.run.smithi026.stdout:(27/30): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi026.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-24T10:31:04.887 INFO:teuthology.orchestra.run.smithi026.stdout:(29/30): librbd1-18.2.2-952.g738c2fd6.el9.x86_6 11 MB/s | 3.0 MB 00:00 2024-03-24T10:31:05.157 INFO:teuthology.orchestra.run.smithi026.stdout:(30/30): librados2-18.2.2-952.g738c2fd6.el9.x86 5.3 MB/s | 3.3 MB 00:00 2024-03-24T10:31:05.161 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:05.162 INFO:teuthology.orchestra.run.smithi026.stdout:Total 13 MB/s | 53 MB 00:04 2024-03-24T10:31:05.162 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:31:05.203 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:31:05.203 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-03-24T10:31:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:05.677 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:31:05.677 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:31:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T10:31:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T10:31:06.428 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:06.428 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:31:06.429 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:31:06.429 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:31:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T10:31:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T10:31:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:07.082 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:31:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-24T10:31:07.211 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-24T10:31:07.266 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-24T10:31:07.567 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:07.567 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 1/32 2024-03-24T10:31:07.567 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2/32 2024-03-24T10:31:07.567 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 3/32 2024-03-24T10:31:07.567 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 4/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 8/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 9/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 12/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 13/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-24T10:31:07.568 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-24T10:31:07.569 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-24T10:31:07.570 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-24T10:31:07.570 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:31:07.570 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-24T10:31:07.570 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 31/32 2024-03-24T10:31:07.646 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-24T10:31:07.677 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:31:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:31:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T10:31:07.895 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:31:07.957 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:31:07.986 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-24T10:31:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 9/32 2024-03-24T10:31:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T10:31:08.164 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:31:08.213 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:31:08.322 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-24T10:31:08.355 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-24T10:31:08.395 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-24T10:31:08.419 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-24T10:31:08.446 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-24T10:31:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-24T10:31:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-24T10:31:08.523 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.524 INFO:teuthology.orchestra.run.smithi031.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:31:08.525 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:08.526 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:08.840 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-24T10:31:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T10:31:08.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-03-24T10:31:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T10:31:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 21/32 2024-03-24T10:31:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-24T10:31:09.434 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-24T10:31:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T10:31:09.484 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:27 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:09.526 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T10:31:09.588 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 25/32 2024-03-24T10:31:09.616 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 26/32 2024-03-24T10:31:09.649 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 40 M 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 166 k 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:09.651 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 41 M 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 165 M 2024-03-24T10:31:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:10.099 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-24T10:31:10.149 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T10:31:10.183 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T10:31:10.325 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-24T10:31:10.500 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): libcephsqlite-18.2.2-952.g738c2fd6.el9.x 295 kB/s | 166 kB 00:00 2024-03-24T10:31:11.786 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:31:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:31:12.370 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T10:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T10:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T10:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:31:15.725 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-test-18.2.2-952.g738c2fd6.el9.x86_6 6.9 MB/s | 40 MB 00:05 2024-03-24T10:31:15.728 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:15.728 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.7 MB/s | 41 MB 00:06 2024-03-24T10:31:15.742 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:15.765 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:15.766 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T10:31:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T10:31:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T10:31:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T10:31:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T10:31:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T10:31:20.892 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-03-24T10:31:22.611 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T10:31:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 1/6 2024-03-24T10:31:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2/6 2024-03-24T10:31:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T10:31:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T10:31:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T10:31:23.377 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:31:23.401 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:23.403 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:31:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:31:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:31:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:23.466 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:23.487 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T10:31:23.487 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:23.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-03-24T10:31:24.334 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:42 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:24.501 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:24.503 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-03-24T10:31:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:31:24.503 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-03-24T10:31:24.503 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 6.5 k 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 2.1 M 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 1.4 M 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 245 k 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.7 M 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 16 M 2024-03-24T10:31:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:31:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:31:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:24.507 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:24.507 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2024-03-24T10:31:24.507 INFO:teuthology.orchestra.run.smithi031.stdout:Install 33 Packages 2024-03-24T10:31:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:24.507 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 29 M 2024-03-24T10:31:24.508 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 104 M 2024-03-24T10:31:24.508 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 1/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 3/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 4/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T10:31:24.552 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 8/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 9/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 12/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 13/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-24T10:31:24.553 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-24T10:31:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 31/32 2024-03-24T10:31:25.067 INFO:teuthology.orchestra.run.smithi031.stdout:(1/33): ceph-18.2.2-952.g738c2fd6.el9.x86_64.rp 15 kB/s | 6.5 kB 00:00 2024-03-24T10:31:25.351 INFO:teuthology.orchestra.run.smithi031.stdout:(2/33): ceph-mgr-18.2.2-952.g738c2fd6.el9.x86_6 2.1 MB/s | 1.4 MB 00:00 2024-03-24T10:31:25.443 INFO:teuthology.orchestra.run.smithi031.stdout:(3/33): ceph-mds-18.2.2-952.g738c2fd6.el9.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-03-24T10:31:25.561 INFO:teuthology.orchestra.run.smithi031.stdout:(4/33): ceph-mgr-modules-core-18.2.2-952.g738c2 2.1 MB/s | 245 kB 00:00 2024-03-24T10:31:25.676 INFO:teuthology.orchestra.run.smithi031.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-24T10:31:25.713 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T10:31:25.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:25.713 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-03-24T10:31:25.713 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.713 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:31:25.714 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:25.715 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:31:25.716 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:31:25.716 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:25.716 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:31:25.760 INFO:teuthology.orchestra.run.smithi031.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 378 kB/s | 32 kB 00:00 2024-03-24T10:31:25.835 INFO:teuthology.orchestra.run.smithi031.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-24T10:31:25.885 INFO:teuthology.orchestra.run.smithi031.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 778 kB/s | 39 kB 00:00 2024-03-24T10:31:26.019 INFO:teuthology.orchestra.run.smithi031.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 241 kB 00:00 2024-03-24T10:31:26.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-03-24T10:31:26.144 INFO:teuthology.orchestra.run.smithi031.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 9.2 MB/s | 1.1 MB 00:00 2024-03-24T10:31:26.195 INFO:teuthology.orchestra.run.smithi031.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T10:31:26.253 INFO:teuthology.orchestra.run.smithi031.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-24T10:31:26.337 INFO:teuthology.orchestra.run.smithi031.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-03-24T10:31:26.537 INFO:teuthology.orchestra.run.smithi031.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 625 kB/s | 125 kB 00:00 2024-03-24T10:31:26.637 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:28 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:31:26.671 INFO:teuthology.orchestra.run.smithi031.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 326 kB/s | 43 kB 00:00 2024-03-24T10:31:26.763 INFO:teuthology.orchestra.run.smithi031.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-24T10:31:26.794 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:31:26.795 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:26.795 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:31:26.795 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:26.795 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:31:26.795 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 40 M 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 166 k 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-24T10:31:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:26.797 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 41 M 2024-03-24T10:31:26.797 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 165 M 2024-03-24T10:31:26.797 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:31:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-24T10:31:26.880 INFO:teuthology.orchestra.run.smithi031.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-24T10:31:26.922 INFO:teuthology.orchestra.run.smithi031.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-03-24T10:31:26.964 INFO:teuthology.orchestra.run.smithi031.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 559 kB/s | 23 kB 00:00 2024-03-24T10:31:27.006 INFO:teuthology.orchestra.run.smithi031.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 465 kB/s | 19 kB 00:00 2024-03-24T10:31:27.049 INFO:teuthology.orchestra.run.smithi031.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 474 kB/s | 20 kB 00:00 2024-03-24T10:31:27.060 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 987 kB/s | 217 kB 00:00 2024-03-24T10:31:27.090 INFO:teuthology.orchestra.run.smithi031.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-24T10:31:27.110 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-24T10:31:27.132 INFO:teuthology.orchestra.run.smithi031.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-24T10:31:27.152 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-24T10:31:27.286 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-24T10:31:27.307 INFO:teuthology.orchestra.run.smithi031.stdout:(25/33): ceph-osd-18.2.2-952.g738c2fd6.el9.x86_ 8.4 MB/s | 16 MB 00:01 2024-03-24T10:31:27.311 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): libcephsqlite-18.2.2-952.g738c2fd6.el9.x 353 kB/s | 166 kB 00:00 2024-03-24T10:31:27.332 INFO:teuthology.orchestra.run.smithi031.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-24T10:31:27.374 INFO:teuthology.orchestra.run.smithi031.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-24T10:31:27.391 INFO:teuthology.orchestra.run.smithi031.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-24T10:31:27.408 INFO:teuthology.orchestra.run.smithi031.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-03-24T10:31:27.468 INFO:teuthology.orchestra.run.smithi031.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-24T10:31:27.509 INFO:teuthology.orchestra.run.smithi031.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-24T10:31:27.535 INFO:teuthology.orchestra.run.smithi031.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-24T10:31:27.967 INFO:teuthology.orchestra.run.smithi031.stdout:(33/33): ceph-mon-18.2.2-952.g738c2fd6.el9.x86_ 1.6 MB/s | 4.7 MB 00:02 2024-03-24T10:31:27.971 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:27.971 INFO:teuthology.orchestra.run.smithi031.stdout:Total 8.4 MB/s | 29 MB 00:03 2024-03-24T10:31:27.986 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:28.397 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:28.398 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:29.729 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T10:31:29.761 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T10:31:29.794 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T10:31:29.897 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T10:31:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T10:31:30.014 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T10:31:30.098 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T10:31:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T10:31:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T10:31:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T10:31:30.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T10:31:30.489 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T10:31:30.556 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T10:31:30.595 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T10:31:30.630 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T10:31:30.660 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T10:31:30.737 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T10:31:30.778 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T10:31:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T10:31:30.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T10:31:30.878 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T10:31:30.913 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T10:31:31.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T10:31:31.051 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T10:31:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T10:31:31.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T10:31:31.406 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T10:31:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 28/33 2024-03-24T10:31:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:31:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:33.116 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:31:33.141 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:33.616 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T10:31:33.638 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T10:31:33.639 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:33.639 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:31:33.639 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:31:33.639 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:31:33.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T10:31:33.872 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T10:31:33.872 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:33.872 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:31:33.872 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:31:33.873 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:31:33.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:33.941 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T10:31:35.632 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T10:31:35.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 1/33 2024-03-24T10:31:35.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 3/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 4/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 5/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 6/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T10:31:35.633 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T10:31:35.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T10:31:35.635 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:31:36.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:31:36.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:31:36.637 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:36.637 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:36.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-03-24T10:31:37.525 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:37.625 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:31:37.679 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:37.680 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:31:37.680 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:37.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-03-24T10:31:38.314 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:56 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:38.463 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:38.464 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:38.464 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:31:38.464 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 224 k 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 224 k 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 215 k 2024-03-24T10:31:38.465 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:38.714 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.2.2-952.g738c2fd6.el9.noarch.rpm 899 kB/s | 224 kB 00:00 2024-03-24T10:31:38.715 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:38.715 INFO:teuthology.orchestra.run.smithi031.stdout:Total 895 kB/s | 224 kB 00:00 2024-03-24T10:31:38.715 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:38.724 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:38.795 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:39.146 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:31:39.228 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:31:39.588 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:31:39.868 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:31:39.868 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:39.868 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:39.868 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:39.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:39.869 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:40.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T10:31:40.635 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:58 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:40.791 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:40.791 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 143 k 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 143 k 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 395 k 2024-03-24T10:31:40.792 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:41.011 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.2.2-952.g738c2fd 654 kB/s | 143 kB 00:00 2024-03-24T10:31:41.011 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:41.011 INFO:teuthology.orchestra.run.smithi031.stdout:Total 649 kB/s | 143 kB 00:00 2024-03-24T10:31:41.011 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:41.019 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:41.019 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:41.072 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:41.072 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:41.210 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-18.2.2-952.g738c2fd6.el9.x86_6 2.8 MB/s | 40 MB 00:14 2024-03-24T10:31:41.212 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:41.213 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.8 MB/s | 41 MB 00:14 2024-03-24T10:31:41.224 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:41.226 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:31:41.247 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:31:41.247 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:31:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:31:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:31:41.309 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:41.309 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:31:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:41.592 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:31:41.592 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:31:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:31:42.195 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T10:31:42.228 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T10:31:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:42.286 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T10:31:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T10:31:42.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-03-24T10:31:43.081 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:43.180 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:31:43.234 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:43.235 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:31:43.236 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:43.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T10:31:43.866 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:44.020 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:44.020 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 1.8 M 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 23 k 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 15 k 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2024-03-24T10:31:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:44.022 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2024-03-24T10:31:44.022 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.6 M 2024-03-24T10:31:44.022 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:44.521 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): ceph-grafana-dashboards-18.2.2-952.g738c 69 kB/s | 23 kB 00:00 2024-03-24T10:31:44.630 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): ceph-prometheus-alerts-18.2.2-952.g738c2 33 kB/s | 15 kB 00:00 2024-03-24T10:31:44.790 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2024-03-24T10:31:44.989 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 526 kB/s | 188 kB 00:00 2024-03-24T10:31:45.847 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): ceph-mgr-dashboard-18.2.2-952.g738c2fd6. 1.1 MB/s | 1.8 MB 00:01 2024-03-24T10:31:45.851 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:45.852 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-03-24T10:31:45.852 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:45.861 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:45.861 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:45.927 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:45.927 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:46.263 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T10:31:46.263 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-24T10:31:46.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-24T10:31:46.322 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T10:31:46.339 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T10:31:46.573 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 4/5 2024-03-24T10:31:46.601 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T10:31:47.086 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T10:31:47.087 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 1/5 2024-03-24T10:31:47.087 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2/5 2024-03-24T10:31:47.087 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T10:31:47.087 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-24T10:31:47.368 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:47.369 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:47.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T10:31:48.158 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:31:48.314 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:31:48.315 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T10:31:48.315 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 1/6 2024-03-24T10:31:48.315 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2/6 2024-03-24T10:31:48.315 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T10:31:48.316 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T10:31:48.316 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T10:31:48.316 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================== 2024-03-24T10:31:48.316 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================== 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 7.4 M 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:31:48.317 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================== 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 37 M 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2024-03-24T10:31:48.318 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:31:48.651 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:31:48.652 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:31:48.652 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:31:48.652 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:31:48.652 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:48.652 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:31:48.836 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-24T10:31:48.870 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-24T10:31:48.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-03-24T10:31:48.911 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-03-24T10:31:48.971 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T10:31:49.145 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T10:31:49.212 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-24T10:31:49.237 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-24T10:31:49.472 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:51 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:31:49.555 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-24T10:31:49.631 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 6.5 k 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 2.1 M 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 1.4 M 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 245 k 2024-03-24T10:31:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.7 M 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 16 M 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:31:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:31:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout:Install 33 Packages 2024-03-24T10:31:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:49.638 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 29 M 2024-03-24T10:31:49.638 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 104 M 2024-03-24T10:31:49.638 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:31:50.063 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-03-24T10:31:50.070 INFO:teuthology.orchestra.run.smithi026.stdout:(1/33): ceph-18.2.2-952.g738c2fd6.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-03-24T10:31:50.088 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 946 kB/s | 803 kB 00:00 2024-03-24T10:31:50.275 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-24T10:31:50.437 INFO:teuthology.orchestra.run.smithi026.stdout:(2/33): ceph-mgr-18.2.2-952.g738c2fd6.el9.x86_6 2.7 MB/s | 1.4 MB 00:00 2024-03-24T10:31:50.487 INFO:teuthology.orchestra.run.smithi026.stdout:(3/33): ceph-mds-18.2.2-952.g738c2fd6.el9.x86_6 3.6 MB/s | 2.1 MB 00:00 2024-03-24T10:31:50.546 INFO:teuthology.orchestra.run.smithi026.stdout:(4/33): ceph-mgr-modules-core-18.2.2-952.g738c2 4.1 MB/s | 245 kB 00:00 2024-03-24T10:31:50.662 INFO:teuthology.orchestra.run.smithi026.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-24T10:31:50.705 INFO:teuthology.orchestra.run.smithi026.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-24T10:31:50.738 INFO:teuthology.orchestra.run.smithi026.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T10:31:50.771 INFO:teuthology.orchestra.run.smithi026.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T10:31:50.814 INFO:teuthology.orchestra.run.smithi026.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T10:31:50.897 INFO:teuthology.orchestra.run.smithi026.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-24T10:31:50.932 INFO:teuthology.orchestra.run.smithi026.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T10:31:50.973 INFO:teuthology.orchestra.run.smithi026.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-24T10:31:51.019 INFO:teuthology.orchestra.run.smithi026.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-24T10:31:51.122 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-9 3.0 MB/s | 7.4 MB 00:02 2024-03-24T10:31:51.124 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:51.125 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-24T10:31:51.165 INFO:teuthology.orchestra.run.smithi026.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-24T10:31:51.168 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:31:51.194 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:31:51.194 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:31:51.490 INFO:teuthology.orchestra.run.smithi026.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2024-03-24T10:31:51.599 INFO:teuthology.orchestra.run.smithi026.stdout:(16/33): ceph-mon-18.2.2-952.g738c2fd6.el9.x86_ 3.1 MB/s | 4.7 MB 00:01 2024-03-24T10:31:51.637 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:31:51.637 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:31:51.691 INFO:teuthology.orchestra.run.smithi026.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-03-24T10:31:51.775 INFO:teuthology.orchestra.run.smithi026.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-24T10:31:52.033 INFO:teuthology.orchestra.run.smithi026.stdout:(19/33): ceph-osd-18.2.2-952.g738c2fd6.el9.x86_ 10 MB/s | 16 MB 00:01 2024-03-24T10:31:52.050 INFO:teuthology.orchestra.run.smithi026.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 64 kB/s | 18 kB 00:00 2024-03-24T10:31:52.084 INFO:teuthology.orchestra.run.smithi026.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 740 kB/s | 358 kB 00:00 2024-03-24T10:31:52.126 INFO:teuthology.orchestra.run.smithi026.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-24T10:31:52.172 INFO:teuthology.orchestra.run.smithi026.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 346 kB/s | 20 kB 00:00 2024-03-24T10:31:52.185 INFO:teuthology.orchestra.run.smithi026.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 791 kB/s | 46 kB 00:00 2024-03-24T10:31:52.226 INFO:teuthology.orchestra.run.smithi026.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-24T10:31:52.260 INFO:teuthology.orchestra.run.smithi026.stdout:(26/33): python3-jaraco-collections-3.0.0-8.el9 102 kB/s | 23 kB 00:00 2024-03-24T10:31:52.293 INFO:teuthology.orchestra.run.smithi026.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-24T10:31:52.310 INFO:teuthology.orchestra.run.smithi026.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-24T10:31:52.344 INFO:teuthology.orchestra.run.smithi026.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-24T10:31:52.350 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:31:52.386 INFO:teuthology.orchestra.run.smithi026.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 714 kB/s | 90 kB 00:00 2024-03-24T10:31:52.411 INFO:teuthology.orchestra.run.smithi026.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-24T10:31:52.429 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T10:31:52.436 INFO:teuthology.orchestra.run.smithi026.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-24T10:31:52.460 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T10:31:52.462 INFO:teuthology.orchestra.run.smithi026.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-24T10:31:52.469 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:52.469 INFO:teuthology.orchestra.run.smithi026.stdout:Total 10 MB/s | 29 MB 00:02 2024-03-24T10:31:52.483 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:31:52.528 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:31:52.528 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:31:52.550 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T10:31:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T10:31:52.772 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T10:31:52.803 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T10:31:52.891 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T10:31:52.904 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:31:52.905 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:31:53.590 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T10:31:53.712 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T10:31:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:31:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T10:31:54.297 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T10:31:54.330 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T10:31:54.427 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T10:31:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T10:31:54.559 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T10:31:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T10:31:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T10:31:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T10:31:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T10:31:54.844 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T10:31:55.037 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T10:31:55.038 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T10:31:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T10:31:55.132 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T10:31:55.166 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T10:31:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T10:31:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T10:31:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T10:31:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T10:31:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T10:31:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T10:31:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T10:31:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T10:31:55.591 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T10:31:55.674 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T10:31:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T10:31:55.949 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T10:31:56.105 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T10:31:56.108 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 28/33 2024-03-24T10:31:56.126 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T10:31:56.128 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T10:31:56.153 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T10:31:56.153 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:56.154 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:31:56.154 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:31:56.154 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:31:56.154 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:31:57.681 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:58.155 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T10:31:58.179 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T10:31:58.180 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:58.180 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:31:58.180 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:31:58.180 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:31:58.180 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T10:31:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T10:31:58.420 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:58.420 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:31:58.420 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:31:58.421 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:31:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:31:58.489 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T10:31:58.596 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 1/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T10:31:58.597 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:31:59.055 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:31:59.056 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:31:59.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2024-03-24T10:31:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:17 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:00.125 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 49 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:32:00.127 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2024-03-24T10:32:00.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:00.129 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2024-03-24T10:32:00.129 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2024-03-24T10:32:00.129 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 1/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 3/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 4/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 5/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 6/33 2024-03-24T10:32:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T10:32:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T10:32:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T10:32:00.418 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 18 kB 00:00 2024-03-24T10:32:00.435 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-24T10:32:00.485 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-24T10:32:00.511 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T10:32:00.536 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T10:32:00.561 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-24T10:32:00.737 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): ceph-mgr-rook-18.2.2-952.g738c2fd6.el9. 110 kB/s | 49 kB 00:00 2024-03-24T10:32:00.811 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-24T10:32:00.920 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 84 kB/s | 32 kB 00:00 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:01.118 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:32:01.119 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:32:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:32:01.121 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:01.121 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:01.442 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-03-24T10:32:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 187 kB/s | 203 kB 00:01 2024-03-24T10:32:02.015 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:04 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:02.116 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:02.170 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:02.171 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:02.171 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:02.179 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 47 kB/s | 59 kB 00:01 2024-03-24T10:32:02.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-03-24T10:32:02.471 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 138 kB/s | 90 kB 00:00 2024-03-24T10:32:02.800 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:04 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:02.953 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:02.953 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:02.953 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 224 k 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 224 k 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 215 k 2024-03-24T10:32:02.954 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:03.271 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 425 kB/s | 1.0 MB 00:02 2024-03-24T10:32:03.275 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:03.275 INFO:teuthology.orchestra.run.smithi031.stdout:Total 666 kB/s | 2.0 MB 00:03 2024-03-24T10:32:03.276 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:03.292 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:03.292 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:03.393 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:03.393 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:03.637 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-18.2.2-952.g738c2fd6.el9.noarch.rpm 328 kB/s | 224 kB 00:00 2024-03-24T10:32:03.638 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:03.639 INFO:teuthology.orchestra.run.smithi026.stdout:Total 327 kB/s | 224 kB 00:00 2024-03-24T10:32:03.639 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:03.645 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:03.645 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:03.648 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:03.648 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:32:04.110 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T10:32:04.165 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T10:32:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:32:04.221 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T10:32:04.256 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T10:32:04.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T10:32:04.397 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T10:32:04.506 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T10:32:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:32:04.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T10:32:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T10:32:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:04.802 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:04.803 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:05.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T10:32:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T10:32:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T10:32:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T10:32:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T10:32:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T10:32:05.581 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:07 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:05.735 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:05.735 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 143 k 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:05.736 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:05.737 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 143 k 2024-03-24T10:32:05.737 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 395 k 2024-03-24T10:32:05.737 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 1/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T10:32:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T10:32:06.401 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-18.2.2-952.g738c2fd 215 kB/s | 143 kB 00:00 2024-03-24T10:32:06.401 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:06.402 INFO:teuthology.orchestra.run.smithi026.stdout:Total 214 kB/s | 143 kB 00:00 2024-03-24T10:32:06.402 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:06.411 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:06.412 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:06.468 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:06.469 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:06.618 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T10:32:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:06.667 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:32:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:32:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:06.669 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:32:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:32:06.693 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:06.693 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:32:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:06.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T10:32:07.456 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:07.614 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 138 k 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:32:07.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.8 M 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2024-03-24T10:32:07.616 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:07.652 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T10:32:07.653 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:07.653 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:07.653 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:07.653 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:07.653 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:07.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-03-24T10:32:07.972 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-24T10:32:08.040 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T10:32:08.189 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 385 kB/s | 58 kB 00:00 2024-03-24T10:32:08.255 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 767 kB/s | 51 kB 00:00 2024-03-24T10:32:08.297 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-24T10:32:08.447 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-mgr-cephadm-18.2.2-952.g738c2fd6.el 206 kB/s | 138 kB 00:00 2024-03-24T10:32:08.453 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:08.453 INFO:teuthology.orchestra.run.smithi031.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-24T10:32:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:08.472 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:08.472 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:08.515 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:10 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:08.563 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:08.563 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:08.615 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:08.669 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:08.670 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:08.670 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:08.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T10:32:08.886 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:09.062 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T10:32:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T10:32:09.298 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:11 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:09.453 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:09.454 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 1.8 M 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 23 k 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 15 k 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:09.455 INFO:teuthology.orchestra.run.smithi026.stdout:Install 5 Packages 2024-03-24T10:32:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:09.456 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-03-24T10:32:09.456 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 8.6 M 2024-03-24T10:32:09.456 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T10:32:09.754 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T10:32:09.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T10:32:09.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T10:32:09.871 INFO:teuthology.orchestra.run.smithi026.stdout:(1/5): ceph-grafana-dashboards-18.2.2-952.g738c 63 kB/s | 23 kB 00:00 2024-03-24T10:32:09.889 INFO:teuthology.orchestra.run.smithi026.stdout:(2/5): ceph-prometheus-alerts-18.2.2-952.g738c2 38 kB/s | 15 kB 00:00 2024-03-24T10:32:10.172 INFO:teuthology.orchestra.run.smithi026.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 102 kB/s | 31 kB 00:00 2024-03-24T10:32:10.272 INFO:teuthology.orchestra.run.smithi026.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 492 kB/s | 188 kB 00:00 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/6 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T10:32:10.583 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T10:32:10.806 INFO:teuthology.orchestra.run.smithi026.stdout:(5/5): ceph-mgr-dashboard-18.2.2-952.g738c2fd6. 1.4 MB/s | 1.8 MB 00:01 2024-03-24T10:32:10.809 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:10.810 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-24T10:32:10.810 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:10.819 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:10.819 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:32:10.875 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:32:10.876 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:10.876 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:10.889 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:10.889 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:11.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-03-24T10:32:11.139 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:11.213 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-24T10:32:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-24T10:32:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T10:32:11.530 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 4/5 2024-03-24T10:32:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T10:32:11.676 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:29 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:11.832 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:11.832 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 844 k 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-03-24T10:32:11.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:11.834 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 922 k 2024-03-24T10:32:11.834 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.7 M 2024-03-24T10:32:11.834 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:11.957 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 79 kB 00:00 2024-03-24T10:32:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T10:32:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 1/5 2024-03-24T10:32:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2/5 2024-03-24T10:32:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T10:32:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-24T10:32:12.307 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.2.2-952.g738c2fd6.el9.x86_6 1.7 MB/s | 844 kB 00:00 2024-03-24T10:32:12.308 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:12.308 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.9 MB/s | 922 kB 00:00 2024-03-24T10:32:12.308 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-24T10:32:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:12.402 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:12.403 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:12.406 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:12.407 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:12.590 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:12.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T10:32:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T10:32:12.800 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T10:32:13.184 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:15 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:13.341 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:13.342 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-03-24T10:32:13.342 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:13.342 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-03-24T10:32:13.342 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 7.4 M 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:32:13.343 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout:=========================================================================================== 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:13.344 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 37 M 2024-03-24T10:32:13.345 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-03-24T10:32:13.345 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:13.551 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T10:32:13.551 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/2 2024-03-24T10:32:13.847 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T10:32:13.847 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:13.848 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:13.848 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T10:32:13.848 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:13.848 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:13.937 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 214 kB/s | 32 kB 00:00 2024-03-24T10:32:13.970 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-24T10:32:14.012 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 846 kB/s | 34 kB 00:00 2024-03-24T10:32:14.071 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T10:32:14.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-03-24T10:32:14.299 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-24T10:32:14.371 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-24T10:32:14.399 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-24T10:32:14.559 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-24T10:32:14.692 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:14.747 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-24T10:32:14.840 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 695 kB/s | 195 kB 00:00 2024-03-24T10:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2024-03-24T10:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2024-03-24T10:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:14.956 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-24T10:32:15.079 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.2.2-952.g738c2fd6.el9.x86_64. 548 kB/s | 127 kB 00:00 2024-03-24T10:32:15.080 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:15.080 INFO:teuthology.orchestra.run.smithi031.stdout:Total 544 kB/s | 127 kB 00:00 2024-03-24T10:32:15.080 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:15.086 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:15.086 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:15.144 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:15.144 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:15.290 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:15.364 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-9 4.7 MB/s | 7.4 MB 00:01 2024-03-24T10:32:15.372 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:15.372 INFO:teuthology.orchestra.run.smithi026.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-24T10:32:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:15.409 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:15.435 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:15.435 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:15.842 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:15.888 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:15.888 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:16.122 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:16.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-03-24T10:32:16.722 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:16.804 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T10:32:16.835 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T10:32:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T10:32:16.937 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:16.972 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T10:32:17.037 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:17.092 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:17.093 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:17.093 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:17.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-03-24T10:32:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T10:32:17.196 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T10:32:17.277 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T10:32:17.729 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:17.884 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:17.884 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:17.884 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-03-24T10:32:17.884 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:17.884 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 31 k 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2024-03-24T10:32:17.885 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 139 k 2024-03-24T10:32:17.886 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T10:32:18.049 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.2.2-952.g738c2fd6.el9.x86_64 190 kB/s | 31 kB 00:00 2024-03-24T10:32:18.050 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:18.050 INFO:teuthology.orchestra.run.smithi031.stdout:Total 188 kB/s | 31 kB 00:00 2024-03-24T10:32:18.050 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:18.053 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:18.054 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:18.079 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T10:32:18.169 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:18.283 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:18.649 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:18.905 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:18.906 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:19.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-03-24T10:32:19.395 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T10:32:19.658 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:19.759 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:19.814 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:19.815 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:19.815 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:19.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-03-24T10:32:20.446 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:20.462 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T10:32:20.486 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T10:32:20.546 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:20.601 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:20.602 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:20.603 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:20.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-03-24T10:32:21.230 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:21.329 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:21.384 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:21.385 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:21.385 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:21.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-03-24T10:32:22.012 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:22.111 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:22.166 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:22.168 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:22.168 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:22.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-03-24T10:32:22.800 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:22.900 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:22.955 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:22.956 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:22.956 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 1/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T10:32:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T10:32:23.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-03-24T10:32:23.437 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:32:23.438 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:32:23.439 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:23.439 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:23.439 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:32:23.439 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:23.439 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:23.593 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:23.692 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:23.749 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:23.749 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-03-24T10:32:23.749 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:23.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-03-24T10:32:23.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-03-24T10:32:24.372 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:26 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:24.381 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:24.527 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:24.528 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:24.528 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 49 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:32:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-03-24T10:32:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:24.531 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-03-24T10:32:24.531 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-03-24T10:32:24.531 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:24.539 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 87 k 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 87 k 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 227 k 2024-03-24T10:32:24.540 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:24.852 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.2.2-952.g738c2fd6.el9.x86_64.rpm 278 kB/s | 87 kB 00:00 2024-03-24T10:32:24.852 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:24.853 INFO:teuthology.orchestra.run.smithi031.stdout:Total 277 kB/s | 87 kB 00:00 2024-03-24T10:32:24.853 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:24.861 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:24.861 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:24.918 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:24.918 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:25.020 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 24 kB 00:00 2024-03-24T10:32:25.047 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-24T10:32:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:25.096 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T10:32:25.113 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-24T10:32:25.138 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-24T10:32:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-24T10:32:25.188 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): ceph-mgr-rook-18.2.2-952.g738c2fd6.el9. 162 kB/s | 49 kB 00:00 2024-03-24T10:32:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-24T10:32:25.282 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2024-03-24T10:32:25.339 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-24T10:32:25.365 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-03-24T10:32:25.406 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-24T10:32:25.526 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:25.607 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-03-24T10:32:25.608 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:25.609 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-24T10:32:25.609 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:25.625 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:25.625 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:25.723 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:25.724 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:25.771 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:26.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-03-24T10:32:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:26.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T10:32:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T10:32:26.557 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T10:32:26.589 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:44 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:26.592 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T10:32:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T10:32:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T10:32:26.747 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:26.748 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:32:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.0 M 2024-03-24T10:32:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 12 M 2024-03-24T10:32:26.749 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:26.848 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T10:32:26.911 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T10:32:27.221 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.2.2-952.g738c2fd6.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-03-24T10:32:27.222 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:27.222 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.3 MB/s | 3.0 MB 00:00 2024-03-24T10:32:27.222 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:27.232 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:27.232 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:27.306 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:27.307 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T10:32:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T10:32:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T10:32:27.807 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T10:32:27.828 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T10:32:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:28.562 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 1/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T10:32:28.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T10:32:28.919 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:29.035 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:29.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:29.035 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:29.035 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:29.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:29.036 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:29.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T10:32:29.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-03-24T10:32:29.745 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:29.863 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 10:30:42 AM UTC. 2024-03-24T10:32:29.899 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 138 k 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:32:29.901 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.8 M 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-03-24T10:32:29.902 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:30.020 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 172 k 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:30.021 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 172 k 2024-03-24T10:32:30.022 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 494 k 2024-03-24T10:32:30.022 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-24T10:32:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.2.2-952.g738c2fd6.el9.x86_64.rpm 758 kB/s | 172 kB 00:00 2024-03-24T10:32:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:Total 753 kB/s | 172 kB 00:00 2024-03-24T10:32:30.250 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-24T10:32:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-24T10:32:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-24T10:32:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-24T10:32:30.316 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-24T10:32:30.467 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-24T10:32:30.500 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-24T10:32:30.578 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:30.584 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-24T10:32:30.684 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): ceph-mgr-cephadm-18.2.2-952.g738c2fd6.el 357 kB/s | 138 kB 00:00 2024-03-24T10:32:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:30.968 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-03-24T10:32:31.001 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-24T10:32:31.026 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-24T10:32:31.030 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:31.030 INFO:teuthology.orchestra.run.smithi026.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-24T10:32:31.043 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:31.053 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:31.054 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:31.145 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:31.150 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:31.261 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:31.261 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:31.261 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-24T10:32:31.261 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:31.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:32:31.262 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-24T10:32:31.458 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:31.488 DEBUG:teuthology.parallel:result is None 2024-03-24T10:32:31.632 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T10:32:31.699 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T10:32:32.255 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T10:32:32.341 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T10:32:32.411 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T10:32:32.440 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T10:32:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T10:32:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/6 2024-03-24T10:32:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T10:32:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T10:32:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T10:32:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:32:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:32:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:33.511 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:33.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-03-24T10:32:34.328 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:36 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:34.486 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 844 k 2024-03-24T10:32:34.487 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 922 k 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.7 M 2024-03-24T10:32:34.488 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:34.631 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 564 kB/s | 79 kB 00:00 2024-03-24T10:32:34.828 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-18.2.2-952.g738c2fd6.el9.x86_6 2.4 MB/s | 844 kB 00:00 2024-03-24T10:32:34.829 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:34.830 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.6 MB/s | 922 kB 00:00 2024-03-24T10:32:34.830 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:34.855 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:34.855 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:34.926 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:34.926 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T10:32:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T10:32:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T10:32:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/2 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:36.339 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:36.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-03-24T10:32:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:37.337 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 127 k 2024-03-24T10:32:37.338 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 456 k 2024-03-24T10:32:37.339 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:37.574 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-18.2.2-952.g738c2fd6.el9.x86_64. 539 kB/s | 127 kB 00:00 2024-03-24T10:32:37.575 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:37.575 INFO:teuthology.orchestra.run.smithi026.stdout:Total 536 kB/s | 127 kB 00:00 2024-03-24T10:32:37.575 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:37.580 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:37.580 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:37.639 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:37.639 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:37.785 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:38.308 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:38.540 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:38.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-03-24T10:32:39.354 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:39.455 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:39.506 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:39.507 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:39.507 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:39.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-03-24T10:32:40.267 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:40.423 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:40.423 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 31 k 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 31 k 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 139 k 2024-03-24T10:32:40.424 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:40.662 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-18.2.2-952.g738c2fd6.el9.x86_64 131 kB/s | 31 kB 00:00 2024-03-24T10:32:40.663 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:40.663 INFO:teuthology.orchestra.run.smithi026.stdout:Total 131 kB/s | 31 kB 00:00 2024-03-24T10:32:40.663 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:40.665 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:40.666 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:40.695 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:40.696 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:41.258 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:41.515 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:41.515 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:41.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-03-24T10:32:42.297 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:42.398 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:42.453 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:42.454 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:42.454 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:42.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-03-24T10:32:43.086 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:43.188 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:43.244 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:43.244 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:43.244 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:43.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-03-24T10:32:43.887 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:43.991 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:44.043 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:44.044 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:44.044 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:44.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-03-24T10:32:44.806 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:46 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:44.907 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:44.960 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:44.961 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:44.962 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:45.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-03-24T10:32:45.604 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:45.704 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:45.759 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:45.760 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:45.760 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:45.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-03-24T10:32:46.389 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:46.487 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T10:32:46.542 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:46.545 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-24T10:32:46.545 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:46.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-03-24T10:32:47.185 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:49 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:47.340 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 87 k 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:47.341 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:47.342 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:47.342 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 87 k 2024-03-24T10:32:47.342 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 227 k 2024-03-24T10:32:47.342 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:47.663 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-18.2.2-952.g738c2fd6.el9.x86_64.rpm 269 kB/s | 87 kB 00:00 2024-03-24T10:32:47.664 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:47.664 INFO:teuthology.orchestra.run.smithi026.stdout:Total 268 kB/s | 87 kB 00:00 2024-03-24T10:32:47.664 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:47.672 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:47.672 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:47.730 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:47.730 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:47.868 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:47.964 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:48.422 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:48.689 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:48.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:48.690 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:48.690 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:48.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:48.690 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:48.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-03-24T10:32:49.492 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:49.649 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:49.649 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:49.650 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:49.651 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:49.651 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.0 M 2024-03-24T10:32:49.651 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 12 M 2024-03-24T10:32:49.651 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:50.083 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-18.2.2-952.g738c2fd6.el9.x86_64.rpm 6.9 MB/s | 3.0 MB 00:00 2024-03-24T10:32:50.084 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:50.084 INFO:teuthology.orchestra.run.smithi026.stdout:Total 6.9 MB/s | 3.0 MB 00:00 2024-03-24T10:32:50.084 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:50.094 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:50.094 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:50.170 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:50.170 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:50.675 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:50.675 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:32:50.675 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:50.676 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:51.790 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:52.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-03-24T10:32:52.645 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:54 ago on Sun 24 Mar 2024 10:30:58 AM UTC. 2024-03-24T10:32:52.797 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-24T10:32:52.797 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 172 k 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 172 k 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 494 k 2024-03-24T10:32:52.798 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-24T10:32:53.003 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-18.2.2-952.g738c2fd6.el9.x86_64.rpm 843 kB/s | 172 kB 00:00 2024-03-24T10:32:53.004 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:32:53.004 INFO:teuthology.orchestra.run.smithi026.stdout:Total 837 kB/s | 172 kB 00:00 2024-03-24T10:32:53.004 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-24T10:32:53.012 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-24T10:32:53.012 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-24T10:32:53.070 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-24T10:32:53.070 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-24T10:32:53.228 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-24T10:32:53.329 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:53.753 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:54.045 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T10:32:54.045 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:54.045 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-24T10:32:54.045 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T10:32:54.046 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:32:54.046 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-24T10:32:54.291 DEBUG:teuthology.parallel:result is None 2024-03-24T10:32:54.292 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:32:54.292 INFO:teuthology.packaging:ref: None 2024-03-24T10:32:54.292 INFO:teuthology.packaging:tag: None 2024-03-24T10:32:54.292 INFO:teuthology.packaging:branch: reef 2024-03-24T10:32:54.292 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:32:54.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T10:32:54.648 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:32:54.669 INFO:teuthology.orchestra.run.smithi026.stdout:18.2.2-952.g738c2fd6.el9 2024-03-24T10:32:54.670 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-952.g738c2fd6.el9 2024-03-24T10:32:54.670 INFO:teuthology.task.install:The correct ceph version 18.2.2-952.g738c2fd6 is installed. 2024-03-24T10:32:54.671 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:32:54.672 INFO:teuthology.packaging:ref: None 2024-03-24T10:32:54.672 INFO:teuthology.packaging:tag: None 2024-03-24T10:32:54.672 INFO:teuthology.packaging:branch: reef 2024-03-24T10:32:54.672 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:32:54.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T10:32:54.991 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:32:55.017 INFO:teuthology.orchestra.run.smithi031.stdout:18.2.2-952.g738c2fd6.el9 2024-03-24T10:32:55.018 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-952.g738c2fd6.el9 2024-03-24T10:32:55.018 INFO:teuthology.task.install:The correct ceph version 18.2.2-952.g738c2fd6 is installed. 2024-03-24T10:32:55.019 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T10:32:55.020 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:32:55.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:32:55.053 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:32:55.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:32:55.092 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T10:32:55.093 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:32:55.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:32:55.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:32:55.193 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:32:55.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:32:55.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:32:55.294 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T10:32:55.295 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:32:55.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:32:55.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:32:55.392 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:32:55.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:32:55.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:32:55.495 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T10:32:55.496 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:32:55.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:32:55.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:32:55.592 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:32:55.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:32:55.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:32:55.689 INFO:teuthology.run_tasks:Running task print... 2024-03-24T10:32:55.699 INFO:teuthology.task.print:**** done install task... 2024-03-24T10:32:55.699 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T10:32:55.899 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T10:32:55.899 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-03-24T10:32:55.902 INFO:tasks.cephadm:Cluster fsid is e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:55.902 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T10:32:55.902 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-24T10:32:55.902 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi026': '172.21.15.26', 'mon.smithi031': '172.21.15.31'} 2024-03-24T10:32:55.903 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T10:32:55.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname $(hostname -s) 2024-03-24T10:32:55.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-03-24T10:32:55.971 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-24T10:32:55.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:32:56.111 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77900/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1855-g21b3bd67', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1855.g21b3bd67'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 04:48:23.862772', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240323.003144-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T10:32:56.343 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 738c2fd67dbc0b17e2e99568a03974712d998229 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-24T10:32:56.344 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm 2024-03-24T10:32:56.345 DEBUG:teuthology.orchestra.run.smithi026:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.219 INFO:teuthology.orchestra.run.smithi026.stdout:-rw-r--r--. 1 ubuntu ubuntu 215105 Mar 24 10:32 /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.220 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 215105 Mar 24 10:32 /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.852 DEBUG:teuthology.orchestra.run.smithi026:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.871 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T10:32:57.897 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-03-24T10:32:57.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-24T10:32:57.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-24T10:32:58.672 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T10:32:58.699 INFO:teuthology.orchestra.run.smithi026.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T10:33:35.588 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)", 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout: "image_id": "68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46", 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout: "repo_digests": [ 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d" 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-24T10:33:35.589 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-24T10:33:45.575 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-24T10:33:45.575 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)", 2024-03-24T10:33:45.575 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46", 2024-03-24T10:33:45.575 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-03-24T10:33:45.576 INFO:teuthology.orchestra.run.smithi031.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d" 2024-03-24T10:33:45.576 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-24T10:33:45.576 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-24T10:33:45.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph 2024-03-24T10:33:45.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-03-24T10:33:45.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /etc/ceph 2024-03-24T10:33:45.699 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-03-24T10:33:45.727 INFO:tasks.cephadm:Writing seed config... 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-24T10:33:45.729 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T10:33:45.730 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T10:33:45.731 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-24T10:33:45.732 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T10:33:45.733 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:33:45.733 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T10:33:45.755 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e07c400a-e9c9-11ee-95cd-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T10:33:45.755 DEBUG:teuthology.orchestra.run.smithi026:mon.smithi026> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service 2024-03-24T10:33:45.797 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T10:33:45.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.26 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:33:45.961 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:33:45.961 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', 'e07c400a-e9c9-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.26', '--skip-admin-label'] 2024-03-24T10:33:45.984 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:33:45.985 INFO:teuthology.orchestra.run.smithi026.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T10:33:45.985 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying podman|docker is present... 2024-03-24T10:33:46.008 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:33:46.008 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying lvm2 is present... 2024-03-24T10:33:46.008 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying time synchronization is in place... 2024-03-24T10:33:46.016 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T10:33:46.016 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T10:33:46.021 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T10:33:46.021 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:33:46.027 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-24T10:33:46.033 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-24T10:33:46.033 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-24T10:33:46.033 INFO:teuthology.orchestra.run.smithi026.stdout:Repeating the final host check... 2024-03-24T10:33:46.058 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:33:46.058 INFO:teuthology.orchestra.run.smithi026.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T10:33:46.058 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl is present 2024-03-24T10:33:46.058 INFO:teuthology.orchestra.run.smithi026.stdout:lvcreate is present 2024-03-24T10:33:46.070 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T10:33:46.070 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T10:33:46.070 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T10:33:46.070 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:33:46.073 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:Host looks OK 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:Cluster fsid: e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:Acquiring lock 140151866397024 on /run/cephadm/e07c400a-e9c9-11ee-95cd-87774f69a715.lock 2024-03-24T10:33:46.080 INFO:teuthology.orchestra.run.smithi026.stdout:Lock 140151866397024 acquired on /run/cephadm/e07c400a-e9c9-11ee-95cd-87774f69a715.lock 2024-03-24T10:33:46.081 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 3300 ... 2024-03-24T10:33:46.081 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 6789 ... 2024-03-24T10:33:46.081 INFO:teuthology.orchestra.run.smithi026.stdout:Base mon IP(s) is [172.21.15.26:3300, 172.21.15.26:6789], mon addrv is [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-24T10:33:46.083 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.26 metric 100 2024-03-24T10:33:46.084 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.26 metric 100 2024-03-24T10:33:46.086 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T10:33:46.086 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T10:33:46.086 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T10:33:46.088 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T10:33:46.088 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T10:33:46.088 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T10:33:46.088 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T10:33:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-03-24T10:33:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T10:33:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-24T10:33:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-24T10:33:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T10:33:46.090 INFO:teuthology.orchestra.run.smithi026.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T10:33:46.091 INFO:teuthology.orchestra.run.smithi026.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:8566210fcee75c6a55e6eeb2f06888d926f27e0de261f0bce516c1cbd24b1704 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying config sha256:68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 2024-03-24T10:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T10:33:47.606 INFO:teuthology.orchestra.run.smithi026.stdout:ceph: stdout ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable) 2024-03-24T10:33:47.606 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph version: ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable) 2024-03-24T10:33:47.606 INFO:teuthology.orchestra.run.smithi026.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T10:33:48.494 INFO:teuthology.orchestra.run.smithi026.stdout:stat: stdout 167 167 2024-03-24T10:33:48.494 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial keys... 2024-03-24T10:33:49.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQCMAQBmufngMBAAN/dSCHAUUhzXiWxxAMkQFQ== 2024-03-24T10:33:50.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQCNAQBm//xqIxAAdyI2SesB0QUcVKx8kEKrzw== 2024-03-24T10:33:51.154 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQCOAQBmtsPGIhAA/4NoC/fMBM1kxopecMBj3A== 2024-03-24T10:33:51.155 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial monmap... 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool for smithi026 [v2:172.21.15.26:3300,v1:172.21.15.26:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = pacific 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: set fsid to e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:33:52.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T10:33:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:33:52.036 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mon... 2024-03-24T10:33:52.907 INFO:teuthology.orchestra.run.smithi026.stdout:create mon.smithi026 on 2024-03-24T10:33:53.125 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T10:33:53.268 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T10:33:53.456 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target. 2024-03-24T10:33:53.456 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target. 2024-03-24T10:33:53.673 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026 2024-03-24T10:33:53.673 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service: Unit ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service not loaded. 2024-03-24T10:33:53.827 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target.wants/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service → /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@.service. 2024-03-24T10:33:54.322 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:33:54.322 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-24T10:33:54.329 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:33:54.329 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:33:54.329 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-24T10:33:54.329 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T10:33:54.330 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon to start... 2024-03-24T10:33:54.330 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon... 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout id: e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout services: 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi026 (age 0.556985s) 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T10:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout data: 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.778+0000 7f56aa479700 1 Processor -- start 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.778+0000 7f56aa479700 1 -- start start 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.778+0000 7f56aa479700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.778+0000 7f56aa479700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a4079e30 con 0x7f56a4106fa0 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.317 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60782/0 (socket says 172.21.15.26:60782) 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a3fff700 1 -- 172.21.15.26:0/196948917 learned_addr learned my addr 172.21.15.26:0/196948917 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a3fff700 1 -- 172.21.15.26:0/196948917 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a41098d0 con 0x7f56a4106fa0 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a3fff700 1 --2- 172.21.15.26:0/196948917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f568c0098a0 tx=0x7f568c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718e578f29f5390f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a2ffd700 1 -- 172.21.15.26:0/196948917 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c004180 con 0x7f56a4106fa0 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a2ffd700 1 -- 172.21.15.26:0/196948917 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c0042e0 con 0x7f56a4106fa0 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.779+0000 7f56a2ffd700 1 -- 172.21.15.26:0/196948917 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c0036a0 con 0x7f56a4106fa0 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 -- 172.21.15.26:0/196948917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 msgr2=0x7f56a4109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 --2- 172.21.15.26:0/196948917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f568c0098a0 tx=0x7f568c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 -- 172.21.15.26:0/196948917 shutdown_connections 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 --2- 172.21.15.26:0/196948917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a4109390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 -- 172.21.15.26:0/196948917 >> 172.21.15.26:0/196948917 conn(0x7f56a4100930 msgr2=0x7f56a4102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.318 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 -- 172.21.15.26:0/196948917 shutdown_connections 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.780+0000 7f56aa479700 1 -- 172.21.15.26:0/196948917 wait complete. 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56aa479700 1 Processor -- start 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56aa479700 1 -- start start 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56aa479700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56aa479700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a4199ef0 con 0x7f56a4106fa0 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60792/0 (socket says 172.21.15.26:60792) 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56a3fff700 1 -- 172.21.15.26:0/2951796192 learned_addr learned my addr 172.21.15.26:0/2951796192 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.781+0000 7f56a3fff700 1 -- 172.21.15.26:0/2951796192 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c009550 con 0x7f56a4106fa0 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a3fff700 1 --2- 172.21.15.26:0/2951796192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56a4079270 tx=0x7f568c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c003ce0 con 0x7f56a4106fa0 2024-03-24T10:33:55.319 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c020070 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a419a0f0 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a419a590 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f568c011630 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f568c022030 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.782+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f568c025d90 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.783+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5684005320 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.785+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f568c025480 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.859+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5684005cc0 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.859+0000 7f56a17fa700 1 -- 172.21.15.26:0/2951796192 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f568c011c10 con 0x7f56a4106fa0 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 msgr2=0x7f56a41999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 --2- 172.21.15.26:0/2951796192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56a4079270 tx=0x7f568c004750 comp rx=0 tx=0).stop 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 shutdown_connections 2024-03-24T10:33:55.320 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 --2- 172.21.15.26:0/2951796192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56a4106fa0 0x7f56a41999b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.321 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 >> 172.21.15.26:0/2951796192 conn(0x7f56a4100930 msgr2=0x7f56a4102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.321 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 shutdown_connections 2024-03-24T10:33:55.321 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:54.860+0000 7f56aa479700 1 -- 172.21.15.26:0/2951796192 wait complete. 2024-03-24T10:33:55.321 INFO:teuthology.orchestra.run.smithi026.stdout:mon is available 2024-03-24T10:33:55.321 INFO:teuthology.orchestra.run.smithi026.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T10:33:56.413 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:56 smithi026 ceph-mon[28829]: from='client.? 172.21.15.26:0/3711754782' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T10:33:56.415 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:56.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.799+0000 7fc109ad4700 1 Processor -- start 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.799+0000 7fc109ad4700 1 -- start start 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.799+0000 7fc109ad4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.799+0000 7fc109ad4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10407a820 con 0x7fc10407bde0 2024-03-24T10:33:56.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60796/0 (socket says 172.21.15.26:60796) 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1037fe700 1 -- 172.21.15.26:0/3756741866 learned_addr learned my addr 172.21.15.26:0/3756741866 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1037fe700 1 -- 172.21.15.26:0/3756741866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10407a960 con 0x7fc10407bde0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1037fe700 1 --2- 172.21.15.26:0/3756741866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098a0 tx=0x7fc0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3934a064778855d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1027fc700 1 -- 172.21.15.26:0/3756741866 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc004180 con 0x7fc10407bde0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1027fc700 1 -- 172.21.15.26:0/3756741866 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0042e0 con 0x7fc10407bde0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.800+0000 7fc1027fc700 1 -- 172.21.15.26:0/3756741866 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc0036a0 con 0x7fc10407bde0 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 -- 172.21.15.26:0/3756741866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 msgr2=0x7fc10407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 --2- 172.21.15.26:0/3756741866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098a0 tx=0x7fc0fc009bb0 comp rx=0 tx=0).stop 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 -- 172.21.15.26:0/3756741866 shutdown_connections 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 --2- 172.21.15.26:0/3756741866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10407a2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 -- 172.21.15.26:0/3756741866 >> 172.21.15.26:0/3756741866 conn(0x7fc104101b40 msgr2=0x7fc104103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 -- 172.21.15.26:0/3756741866 shutdown_connections 2024-03-24T10:33:56.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.801+0000 7fc109ad4700 1 -- 172.21.15.26:0/3756741866 wait complete. 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc109ad4700 1 Processor -- start 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc109ad4700 1 -- start start 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc109ad4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc109ad4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10419c070 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc1037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc1037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60798/0 (socket says 172.21.15.26:60798) 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc1037fe700 1 -- 172.21.15.26:0/3711754782 learned_addr learned my addr 172.21.15.26:0/3711754782 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc1037fe700 1 -- 172.21.15.26:0/3711754782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc009550 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.802+0000 7fc1037fe700 1 --2- 172.21.15.26:0/3711754782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc10407b350 tx=0x7fc0fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc003ce0 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc020070 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc011630 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10419c270 con 0x7fc10407bde0 2024-03-24T10:33:56.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10419c710 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc0fc022030 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.803+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc0fc025d40 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.804+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1041959c0 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.806+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fc0fc025480 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.875+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc10402cf10 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc0fc011c20 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc100ff9700 1 -- 172.21.15.26:0/3711754782 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fc0fc036020 con 0x7fc10407bde0 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 msgr2=0x7fc10419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc109ad4700 1 --2- 172.21.15.26:0/3711754782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc10407b350 tx=0x7fc0fc004750 comp rx=0 tx=0).stop 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 shutdown_connections 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc109ad4700 1 --2- 172.21.15.26:0/3711754782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc10407bde0 0x7fc10419bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.879+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 >> 172.21.15.26:0/3711754782 conn(0x7fc104101b40 msgr2=0x7fc104103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:56.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.880+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 shutdown_connections 2024-03-24T10:33:56.427 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:55.880+0000 7fc109ad4700 1 -- 172.21.15.26:0/3711754782 wait complete. 2024-03-24T10:33:56.427 INFO:teuthology.orchestra.run.smithi026.stdout:Generating new minimal ceph.conf... 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa926581700 1 Processor -- start 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa926581700 1 -- start start 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa926581700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa926581700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920106f00 con 0x7fa920106510 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60808/0 (socket says 172.21.15.26:60808) 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa91ffff700 1 -- 172.21.15.26:0/738242763 learned_addr learned my addr 172.21.15.26:0/738242763 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.816+0000 7fa91ffff700 1 -- 172.21.15.26:0/738242763 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9201076c0 con 0x7fa920106510 2024-03-24T10:33:57.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa91ffff700 1 --2- 172.21.15.26:0/738242763 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa9100098a0 tx=0x7fa910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e7805ee39f82bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa91effd700 1 -- 172.21.15.26:0/738242763 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910004180 con 0x7fa920106510 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa91effd700 1 -- 172.21.15.26:0/738242763 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa9100042e0 con 0x7fa920106510 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa91effd700 1 -- 172.21.15.26:0/738242763 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910011a70 con 0x7fa920106510 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 -- 172.21.15.26:0/738242763 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 msgr2=0x7fa920106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 --2- 172.21.15.26:0/738242763 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa9100098a0 tx=0x7fa910009bb0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 -- 172.21.15.26:0/738242763 shutdown_connections 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 --2- 172.21.15.26:0/738242763 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa920106930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 -- 172.21.15.26:0/738242763 >> 172.21.15.26:0/738242763 conn(0x7fa920101b10 msgr2=0x7fa920103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 -- 172.21.15.26:0/738242763 shutdown_connections 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.817+0000 7fa926581700 1 -- 172.21.15.26:0/738242763 wait complete. 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa926581700 1 Processor -- start 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa926581700 1 -- start start 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa926581700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa926581700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92019a790 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60810/0 (socket says 172.21.15.26:60810) 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.818+0000 7fa91ffff700 1 -- 172.21.15.26:0/512632991 learned_addr learned my addr 172.21.15.26:0/512632991 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91ffff700 1 -- 172.21.15.26:0/512632991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910009550 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91ffff700 1 --2- 172.21.15.26:0/512632991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa910011710 tx=0x7fa9100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910003b10 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa91001b8c0 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92019a990 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910019770 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92019ae30 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa910019ca0 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.819+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa91001a710 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.820+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9200623e0 con 0x7fa920106510 2024-03-24T10:33:57.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.822+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa910029060 con 0x7fa920106510 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.893+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa92010a8f0 con 0x7fa920106510 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.893+0000 7fa91d7fa700 1 -- 172.21.15.26:0/512632991 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fa91001f070 con 0x7fa920106510 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 msgr2=0x7fa92019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 --2- 172.21.15.26:0/512632991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa910011710 tx=0x7fa9100047c0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 shutdown_connections 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 --2- 172.21.15.26:0/512632991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa920106510 0x7fa92019a250 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 >> 172.21.15.26:0/512632991 conn(0x7fa920101b10 msgr2=0x7fa9201026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 shutdown_connections 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:33:56.894+0000 7fa926581700 1 -- 172.21.15.26:0/512632991 wait complete. 2024-03-24T10:33:57.405 INFO:teuthology.orchestra.run.smithi026.stdout:Restarting the monitor... 2024-03-24T10:33:57.419 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:57 smithi026 ceph-mon[28829]: from='client.? 172.21.15.26:0/512632991' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:57.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:57 smithi026 systemd[1]: Stopping Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:33:57.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[28824]: 2024-03-24T10:33:57.655+0000 7f17aca91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi026 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:33:57.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[28824]: 2024-03-24T10:33:57.655+0000 7f17aca91700 -1 mon.smithi026@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T10:33:58.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:57 smithi026 podman[29084]: 2024-03-24 10:33:57.861540137 +0000 UTC m=+0.297586619 container died eb9e186003fda500d710a9885c0017c558e547add429e6b5f35d6a1229379c20 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-738c2fd) 2024-03-24T10:33:59.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:59 smithi026 podman[29084]: 2024-03-24 10:33:59.415698984 +0000 UTC m=+1.851745467 container cleanup eb9e186003fda500d710a9885c0017c558e547add429e6b5f35d6a1229379c20 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T10:33:59.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:33:59 smithi026 bash[29084]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026 2024-03-24T10:34:01.568 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:01 smithi026 podman[29098]: 2024-03-24 10:34:01.313558624 +0000 UTC m=+3.450201961 container remove eb9e186003fda500d710a9885c0017c558e547add429e6b5f35d6a1229379c20 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T10:34:01.568 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:01 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service: Deactivated successfully. 2024-03-24T10:34:01.568 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:01 smithi026 systemd[1]: Stopped Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:34:01.568 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:01 smithi026 systemd[1]: Starting Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:34:01.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:01 smithi026 podman[29191]: 2024-03-24 10:34:01.61486425 +0000 UTC m=+0.017836521 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph:reef 2024-03-24T10:34:03.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:03 smithi026 podman[29191]: 2024-03-24 10:34:03.034166747 +0000 UTC m=+1.437139059 container create dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T10:34:05.142 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 podman[29191]: 2024-03-24 10:34:05.034216327 +0000 UTC m=+3.437188600 container init dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 podman[29191]: 2024-03-24 10:34:05.03853721 +0000 UTC m=+3.441509484 container start dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable), process ceph-mon, pid 2 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: pidfile_write: ignore empty --pid-file 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: load: jerasure load: lrc 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: RocksDB version: 7.9.2 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Git sha 0 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Compile date 2024-03-23 03:05:26 2024-03-24T10:34:05.143 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: DB SUMMARY 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: DB Session ID: GE5N0BVR0E7ZZDVD9Q3O 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: CURRENT file: CURRENT 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi026/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi026/store.db: 000009.log size: 98465 ; 2024-03-24T10:34:05.144 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.error_if_exists: 0 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.create_if_missing: 0 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.paranoid_checks: 1 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.env: 0x55911f558720 2024-03-24T10:34:05.145 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.info_log: 0x55912273b380 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.statistics: (nil) 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.use_fsync: 0 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_log_file_size: 0 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T10:34:05.146 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_fallocate: 1 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.use_direct_reads: 0 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T10:34:05.147 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.db_log_dir: 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.wal_dir: 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T10:34:05.148 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.write_buffer_manager: 0x5591219b4140 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T10:34:05.149 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T10:34:05.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.unordered_write: 0 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T10:34:05.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.row_cache: None 2024-03-24T10:34:05.152 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.wal_filter: None 2024-03-24T10:34:05.152 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T10:34:05.152 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T10:34:05.154 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.two_write_queues: 0 2024-03-24T10:34:05.154 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T10:34:05.155 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.wal_compression: 0 2024-03-24T10:34:05.155 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.atomic_flush: 0 2024-03-24T10:34:05.155 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T10:34:05.155 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T10:34:05.155 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.log_readahead_size: 0 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T10:34:05.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_background_jobs: 2 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_background_compactions: -1 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_subcompactions: 1 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T10:34:05.157 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_open_files: -1 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T10:34:05.158 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_background_flushes: -1 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Compression algorithms supported: 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kZSTD supported: 0 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kXpressCompression supported: 0 2024-03-24T10:34:05.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kZlibCompression supported: 1 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kSnappyCompression supported: 1 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kLZ4Compression supported: 1 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: kBZip2Compression supported: 0 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T10:34:05.161 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T10:34:05.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000010 2024-03-24T10:34:05.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T10:34:05.162 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T10:34:05.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.merge_operator: 2024-03-24T10:34:05.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_filter: None 2024-03-24T10:34:05.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T10:34:05.163 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55912273b4a0) 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T10:34:05.164 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_type: 0 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_index_type: 0 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_shortening: 1 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T10:34:05.165 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: checksum: 4 2024-03-24T10:34:05.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: no_block_cache: 0 2024-03-24T10:34:05.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache: 0x559121a551f0 2024-03-24T10:34:05.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-03-24T10:34:05.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_options: 2024-03-24T10:34:05.166 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: capacity : 536870912 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: num_shard_bits : 4 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: strict_capacity_limit : 0 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_cache_compressed: (nil) 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: persistent_cache: (nil) 2024-03-24T10:34:05.167 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size: 4096 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_size_deviation: 10 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_restart_interval: 16 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: index_block_restart_interval: 1 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: metadata_block_size: 4096 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: partition_filters: 0 2024-03-24T10:34:05.168 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: use_delta_encoding: 1 2024-03-24T10:34:05.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: filter_policy: bloomfilter 2024-03-24T10:34:05.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: whole_key_filtering: 1 2024-03-24T10:34:05.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: verify_compression: 0 2024-03-24T10:34:05.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-03-24T10:34:05.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: format_version: 5 2024-03-24T10:34:05.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_index_compression: 1 2024-03-24T10:34:05.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: block_align: 0 2024-03-24T10:34:05.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: max_auto_readahead_size: 262144 2024-03-24T10:34:05.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: prepopulate_block_cache: 0 2024-03-24T10:34:05.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: initial_auto_readahead_size: 8192 2024-03-24T10:34:05.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T10:34:05.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T10:34:05.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T10:34:05.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression: NoCompression 2024-03-24T10:34:05.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T10:34:05.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T10:34:05.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T10:34:05.172 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.num_levels: 7 2024-03-24T10:34:05.173 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T10:34:05.173 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T10:34:05.174 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T10:34:05.174 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T10:34:05.174 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T10:34:05.185 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:34:05.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T10:34:05.187 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T10:34:05.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T10:34:05.189 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T10:34:05.190 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T10:34:05.191 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.table_properties_collectors: 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.inplace_update_support: 0 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T10:34:05.192 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.bloom_locality: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.max_successive_merges: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T10:34:05.193 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.ttl: 2592000 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enable_blob_files: false 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.min_blob_size: 0 2024-03-24T10:34:05.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T10:34:05.195 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi026/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b1364562-d126-4fc9-8b0a-e2475e9fcb9d 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276445070463, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T10:34:05.196 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276445071690, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93742, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91776, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84340, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711276445, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b1364562-d126-4fc9-8b0a-e2475e9fcb9d", "db_session_id": "GE5N0BVR0E7ZZDVD9Q3O", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276445071760, "job": 1, "event": "recovery_finished"} 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi026/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559121af2000 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: DB pointer 0x559121ae0000 2024-03-24T10:34:05.197 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T10:34:05.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T10:34:05.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** DB Stats ** 2024-03-24T10:34:05.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:34:05.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T10:34:05.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-24T10:34:05.199 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: L0 2/0 93.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Sum 2/0 93.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T10:34:05.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Cumulative compaction: 0.00 GB write, 26.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Interval compaction: 0.00 GB write, 26.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Block cache BinnedLRUCache@0x559121a551f0#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: starting mon.smithi026 rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi026 fsid e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???) e1 preinit fsid e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).mds e1 new map 2024-03-24T10:34:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).mds e1 print_map 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: e1 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: legacy client fscid: -1 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout: No filesystems configured 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mon.smithi026 is new leader, mons smithi026 in quorum (ranks 0) 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: monmap e1: 1 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T10:34:05.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: fsmap 2024-03-24T10:34:05.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T10:34:05.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 ceph-mon[29210]: mgrmap e1: no daemons active 2024-03-24T10:34:05.922 INFO:teuthology.orchestra.run.smithi026.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T10:34:06.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 bash[29191]: dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 2024-03-24T10:34:06.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:05 smithi026 systemd[1]: Started Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:34:09.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.858+0000 7fe27b05d700 1 Processor -- start 2024-03-24T10:34:09.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.858+0000 7fe27b05d700 1 -- start start 2024-03-24T10:34:09.422 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.858+0000 7fe27b05d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.858+0000 7fe27b05d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe274106f40 con 0x7fe274106550 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe278df9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe278df9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47150/0 (socket says 172.21.15.26:47150) 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe278df9700 1 -- 172.21.15.26:0/1354354635 learned_addr learned my addr 172.21.15.26:0/1354354635 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe278df9700 1 -- 172.21.15.26:0/1354354635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2741076b0 con 0x7fe274106550 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe278df9700 1 --2- 172.21.15.26:0/1354354635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c57c2e13a1828d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe2737fe700 1 -- 172.21.15.26:0/1354354635 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe268004180 con 0x7fe274106550 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe2737fe700 1 -- 172.21.15.26:0/1354354635 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe2680042e0 con 0x7fe274106550 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.859+0000 7fe2737fe700 1 -- 172.21.15.26:0/1354354635 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe268011a70 con 0x7fe274106550 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 -- 172.21.15.26:0/1354354635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 msgr2=0x7fe274106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 --2- 172.21.15.26:0/1354354635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe268009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 -- 172.21.15.26:0/1354354635 shutdown_connections 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 --2- 172.21.15.26:0/1354354635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe274106970 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 -- 172.21.15.26:0/1354354635 >> 172.21.15.26:0/1354354635 conn(0x7fe274101b10 msgr2=0x7fe274103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 -- 172.21.15.26:0/1354354635 shutdown_connections 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.860+0000 7fe27b05d700 1 -- 172.21.15.26:0/1354354635 wait complete. 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe27b05d700 1 Processor -- start 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe27b05d700 1 -- start start 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe27b05d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe27b05d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27419a710 con 0x7fe274106550 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe278df9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe278df9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47152/0 (socket says 172.21.15.26:47152) 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe278df9700 1 -- 172.21.15.26:0/72698952 learned_addr learned my addr 172.21.15.26:0/72698952 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.861+0000 7fe278df9700 1 -- 172.21.15.26:0/72698952 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe268009550 con 0x7fe274106550 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe278df9700 1 --2- 172.21.15.26:0/72698952 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe268011710 tx=0x7fe2680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe268003b10 con 0x7fe274106550 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe26801b8c0 con 0x7fe274106550 2024-03-24T10:34:09.424 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27419a910 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe268019770 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27419adb0 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe268019cf0 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.862+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe26801a710 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.863+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2740623e0 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.865+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe268029060 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.935+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe27419da60 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.937+0000 7fe271ffb700 1 -- 172.21.15.26:0/72698952 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe26801f070 con 0x7fe274106550 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 msgr2=0x7fe27419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 --2- 172.21.15.26:0/72698952 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe268011710 tx=0x7fe2680047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 shutdown_connections 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 --2- 172.21.15.26:0/72698952 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe274106550 0x7fe27419a1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 >> 172.21.15.26:0/72698952 conn(0x7fe274101b10 msgr2=0x7fe274102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.425 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 shutdown_connections 2024-03-24T10:34:09.426 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:08.938+0000 7fe27b05d700 1 -- 172.21.15.26:0/72698952 wait complete. 2024-03-24T10:34:09.426 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T10:34:09.439 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:34:09.439 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mgr... 2024-03-24T10:34:09.440 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T10:34:09.440 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T10:34:09.440 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-24T10:34:09.691 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mgr.smithi026.bbjcru 2024-03-24T10:34:09.692 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mgr.smithi026.bbjcru.service: Unit ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mgr.smithi026.bbjcru.service not loaded. 2024-03-24T10:34:09.843 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715.target.wants/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mgr.smithi026.bbjcru.service → /etc/systemd/system/ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@.service. 2024-03-24T10:34:09.950 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:09 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/72698952' entity='client.admin' 2024-03-24T10:34:10.358 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:34:10.358 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-24T10:34:10.365 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:34:10.366 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:34:10.366 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-24T10:34:10.366 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T10:34:10.374 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:34:10.374 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-24T10:34:10.380 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:34:10.380 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:34:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-24T10:34:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-24T10:34:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr to start... 2024-03-24T10:34:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr... 2024-03-24T10:34:11.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:11 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/4050907859' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:34:11.439 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:11.439 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:11.439 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "e07c400a-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:34:11.439 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:34:11.441 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:34:11.449 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:34:11.450 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:33:54.305662+0000", 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.822+0000 7fca07276700 1 Processor -- start 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca07276700 1 -- start start 2024-03-24T10:34:11.451 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca07276700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca07276700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000745f0 con 0x7fca00106f30 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca05012700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca05012700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47182/0 (socket says 172.21.15.26:47182) 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca05012700 1 -- 172.21.15.26:0/4002685331 learned_addr learned my addr 172.21.15.26:0/4002685331 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.823+0000 7fca05012700 1 -- 172.21.15.26:0/4002685331 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca00109860 con 0x7fca00106f30 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fca05012700 1 --2- 172.21.15.26:0/4002685331 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0098a0 tx=0x7fc9fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e045a3827721cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fc9f3fff700 1 -- 172.21.15.26:0/4002685331 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc004180 con 0x7fca00106f30 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fc9f3fff700 1 -- 172.21.15.26:0/4002685331 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc9fc0042e0 con 0x7fca00106f30 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fc9f3fff700 1 -- 172.21.15.26:0/4002685331 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc011a70 con 0x7fca00106f30 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fca07276700 1 -- 172.21.15.26:0/4002685331 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 msgr2=0x7fca00109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.824+0000 7fca07276700 1 --2- 172.21.15.26:0/4002685331 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0098a0 tx=0x7fc9fc009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- 172.21.15.26:0/4002685331 shutdown_connections 2024-03-24T10:34:11.452 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 --2- 172.21.15.26:0/4002685331 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00109320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- 172.21.15.26:0/4002685331 >> 172.21.15.26:0/4002685331 conn(0x7fca001009c0 msgr2=0x7fca00102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- 172.21.15.26:0/4002685331 shutdown_connections 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- 172.21.15.26:0/4002685331 wait complete. 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 Processor -- start 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- start start 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.825+0000 7fca07276700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca00106ca0 con 0x7fca00106f30 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca05012700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca05012700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47184/0 (socket says 172.21.15.26:47184) 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca05012700 1 -- 172.21.15.26:0/4050907859 learned_addr learned my addr 172.21.15.26:0/4050907859 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca05012700 1 -- 172.21.15.26:0/4050907859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009550 con 0x7fca00106f30 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca05012700 1 --2- 172.21.15.26:0/4050907859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc011a20 tx=0x7fc9fc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc003b10 con 0x7fca00106f30 2024-03-24T10:34:11.453 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc9fc01b8c0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca00104ee0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc0237b0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca00105380 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc9fc023d30 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.826+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc9fc019710 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.827+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0010a5e0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.829+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fc9fc01ba30 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.903+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fca001059e0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.903+0000 7fc9f27fc700 1 -- 172.21.15.26:0/4050907859 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fc9fc01bbd0 con 0x7fca00106f30 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 msgr2=0x7fca00106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 --2- 172.21.15.26:0/4050907859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc011a20 tx=0x7fc9fc0047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:11.454 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 shutdown_connections 2024-03-24T10:34:11.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 --2- 172.21.15.26:0/4050907859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca00106f30 0x7fca00106760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:11.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 >> 172.21.15.26:0/4050907859 conn(0x7fca001009c0 msgr2=0x7fca001014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:11.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 shutdown_connections 2024-03-24T10:34:11.455 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:10.904+0000 7fca07276700 1 -- 172.21.15.26:0/4050907859 wait complete. 2024-03-24T10:34:11.455 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (1/15)... 2024-03-24T10:34:14.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:13 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3192753868' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:34:14.628 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:14.628 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "e07c400a-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-24T10:34:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:34:14.630 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:34:14.644 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:34:14.644 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.645 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:14.646 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:33:54.305662+0000", 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.835+0000 7f9995eb1700 1 Processor -- start 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.835+0000 7f9995eb1700 1 -- start start 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.835+0000 7f9995eb1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.835+0000 7f9995eb1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990105db0 con 0x7f99901053c0 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50304/0 (socket says 172.21.15.26:50304) 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998f7fe700 1 -- 172.21.15.26:0/1288768244 learned_addr learned my addr 172.21.15.26:0/1288768244 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998f7fe700 1 -- 172.21.15.26:0/1288768244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99901065c0 con 0x7f99901053c0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998f7fe700 1 --2- 172.21.15.26:0/1288768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f9980009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ec2156d9549f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998e7fc700 1 -- 172.21.15.26:0/1288768244 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980004180 con 0x7f99901053c0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998e7fc700 1 -- 172.21.15.26:0/1288768244 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f99800042e0 con 0x7f99901053c0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.836+0000 7f998e7fc700 1 -- 172.21.15.26:0/1288768244 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980011a70 con 0x7f99901053c0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 -- 172.21.15.26:0/1288768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 msgr2=0x7f99901057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 --2- 172.21.15.26:0/1288768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f9980009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 -- 172.21.15.26:0/1288768244 shutdown_connections 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 --2- 172.21.15.26:0/1288768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f99901057e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 -- 172.21.15.26:0/1288768244 >> 172.21.15.26:0/1288768244 conn(0x7f99901009c0 msgr2=0x7f9990102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 -- 172.21.15.26:0/1288768244 shutdown_connections 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.837+0000 7f9995eb1700 1 -- 172.21.15.26:0/1288768244 wait complete. 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f9995eb1700 1 Processor -- start 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f9995eb1700 1 -- start start 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f9995eb1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f9995eb1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999019c990 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f998f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f998f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50306/0 (socket says 172.21.15.26:50306) 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.838+0000 7f998f7fe700 1 -- 172.21.15.26:0/3192753868 learned_addr learned my addr 172.21.15.26:0/3192753868 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998f7fe700 1 -- 172.21.15.26:0/3192753868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980009550 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998f7fe700 1 --2- 172.21.15.26:0/3192753868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9980011710 tx=0x7f99800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980003b10 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f998001b8c0 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980019710 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999019cb90 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999019d030 con 0x7f99901053c0 2024-03-24T10:34:14.649 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9980019bc0 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.839+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f998001a710 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.840+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99900623e0 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.842+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f998001ba30 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.915+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9990073710 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.915+0000 7f998cff9700 1 -- 172.21.15.26:0/3192753868 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9980030090 con 0x7f99901053c0 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.916+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 msgr2=0x7f999019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.916+0000 7f9995eb1700 1 --2- 172.21.15.26:0/3192753868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9980011710 tx=0x7f99800047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.916+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 shutdown_connections 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.916+0000 7f9995eb1700 1 --2- 172.21.15.26:0/3192753868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99901053c0 0x7f999019c450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.916+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 >> 172.21.15.26:0/3192753868 conn(0x7f99901009c0 msgr2=0x7f999018e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.917+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 shutdown_connections 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:13.917+0000 7f9995eb1700 1 -- 172.21.15.26:0/3192753868 wait complete. 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (2/15)... 2024-03-24T10:34:17.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:17 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2780777418' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "e07c400a-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:34:17.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:17.787 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:34:17.787 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:34:17.787 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:34:17.797 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:34:17.797 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:34:17.797 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:34:17.797 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:34:17.798 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:34:17.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:33:54.305662+0000", 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f04529f0700 1 Processor -- start 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f04529f0700 1 -- start start 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f04529f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:17.800 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f04529f0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0745b0 con 0x7f044c106f70 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50320/0 (socket says 172.21.15.26:50320) 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044bfff700 1 -- 172.21.15.26:0/3234465929 learned_addr learned my addr 172.21.15.26:0/3234465929 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044bfff700 1 -- 172.21.15.26:0/3234465929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c1098a0 con 0x7f044c106f70 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044bfff700 1 --2- 172.21.15.26:0/3234465929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f043c0098a0 tx=0x7f043c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28d187f9055bbd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.067+0000 7f044affd700 1 -- 172.21.15.26:0/3234465929 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c004180 con 0x7f044c106f70 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f044affd700 1 -- 172.21.15.26:0/3234465929 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f043c0042e0 con 0x7f044c106f70 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f044affd700 1 -- 172.21.15.26:0/3234465929 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c011a70 con 0x7f044c106f70 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 -- 172.21.15.26:0/3234465929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 msgr2=0x7f044c109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 --2- 172.21.15.26:0/3234465929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f043c0098a0 tx=0x7f043c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 -- 172.21.15.26:0/3234465929 shutdown_connections 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 --2- 172.21.15.26:0/3234465929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c109360 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:17.801 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 -- 172.21.15.26:0/3234465929 >> 172.21.15.26:0/3234465929 conn(0x7f044c1009c0 msgr2=0x7f044c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 -- 172.21.15.26:0/3234465929 shutdown_connections 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.068+0000 7f04529f0700 1 -- 172.21.15.26:0/3234465929 wait complete. 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f04529f0700 1 Processor -- start 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f04529f0700 1 -- start start 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f04529f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f04529f0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c106ce0 con 0x7f044c106f70 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f044bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:17.802 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f044bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50328/0 (socket says 172.21.15.26:50328) 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.069+0000 7f044bfff700 1 -- 172.21.15.26:0/2780777418 learned_addr learned my addr 172.21.15.26:0/2780777418 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f044bfff700 1 -- 172.21.15.26:0/2780777418 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c009550 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f044bfff700 1 --2- 172.21.15.26:0/2780777418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f043c011710 tx=0x7f043c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c003b10 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c104f20 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f043c01b8c0 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c0197e0 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c1053c0 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f043c019d20 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.070+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f043c01a750 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.071+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c10a5f0 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.075+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f043c029060 con 0x7f044c106f70 2024-03-24T10:34:17.803 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.147+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f044c10a7e0 con 0x7f044c106f70 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.147+0000 7f04497fa700 1 -- 172.21.15.26:0/2780777418 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f043c01f070 con 0x7f044c106f70 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.148+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 msgr2=0x7f044c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.148+0000 7f04529f0700 1 --2- 172.21.15.26:0/2780777418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f043c011710 tx=0x7f043c0047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.148+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 shutdown_connections 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.148+0000 7f04529f0700 1 --2- 172.21.15.26:0/2780777418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f044c106f70 0x7f044c1067a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.148+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 >> 172.21.15.26:0/2780777418 conn(0x7f044c1009c0 msgr2=0x7f044c1014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.149+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 shutdown_connections 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:17.149+0000 7f04529f0700 1 -- 172.21.15.26:0/2780777418 wait complete. 2024-03-24T10:34:17.804 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (3/15)... 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: Activating manager daemon smithi026.bbjcru 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: mgrmap e2: smithi026.bbjcru(active, starting, since 0.00283998s) 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:34:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi026.bbjcru", "id": "smithi026.bbjcru"}]: dispatch 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: Manager daemon smithi026.bbjcru is now available 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/trash_purge_schedule"}]: dispatch 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:18.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:18 smithi026 ceph-mon[29210]: from='mgr.14100 172.21.15.26:0/2641443341' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:19.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:19 smithi026 ceph-mon[29210]: mgrmap e3: smithi026.bbjcru(active, since 1.00672s) 2024-03-24T10:34:21.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:21.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:21.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "e07c400a-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:34:21.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:34:21.108 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "smithi026" 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:34:21.110 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:34:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:34:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:33:54.305662+0000", 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbd007d700 1 Processor -- start 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbd007d700 1 -- start start 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbd007d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbd007d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8074570 con 0x7fdbc8104da0 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbcde19700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbcde19700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50384/0 (socket says 172.21.15.26:50384) 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.197+0000 7fdbcde19700 1 -- 172.21.15.26:0/2691168840 learned_addr learned my addr 172.21.15.26:0/2691168840 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.198+0000 7fdbcde19700 1 -- 172.21.15.26:0/2691168840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8105700 con 0x7fdbc8104da0 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.198+0000 7fdbcde19700 1 --2- 172.21.15.26:0/2691168840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40098a0 tx=0x7fdbc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33eacb7ba7ffbcc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.198+0000 7fdbcce17700 1 -- 172.21.15.26:0/2691168840 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc4004180 con 0x7fdbc8104da0 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.198+0000 7fdbcce17700 1 -- 172.21.15.26:0/2691168840 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdbc40042e0 con 0x7fdbc8104da0 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.198+0000 7fdbcce17700 1 -- 172.21.15.26:0/2691168840 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc4011a70 con 0x7fdbc8104da0 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 -- 172.21.15.26:0/2691168840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 msgr2=0x7fdbc81051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2691168840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40098a0 tx=0x7fdbc4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 -- 172.21.15.26:0/2691168840 shutdown_connections 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2691168840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc81051c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 -- 172.21.15.26:0/2691168840 >> 172.21.15.26:0/2691168840 conn(0x7fdbc81009c0 msgr2=0x7fdbc8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 -- 172.21.15.26:0/2691168840 shutdown_connections 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.199+0000 7fdbd007d700 1 -- 172.21.15.26:0/2691168840 wait complete. 2024-03-24T10:34:21.114 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbd007d700 1 Processor -- start 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbd007d700 1 -- start start 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbd007d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbd007d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc819c930 con 0x7fdbc8104da0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbcde19700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbcde19700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50388/0 (socket says 172.21.15.26:50388) 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbcde19700 1 -- 172.21.15.26:0/2400404149 learned_addr learned my addr 172.21.15.26:0/2400404149 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbcde19700 1 -- 172.21.15.26:0/2400404149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4009550 con 0x7fdbc8104da0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbcde19700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4011710 tx=0x7fdbc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.200+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc4003b10 con 0x7fdbc8104da0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdbc401b8c0 con 0x7fdbc8104da0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc819cb30 con 0x7fdbc8104da0 2024-03-24T10:34:21.115 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc40197e0 con 0x7fdbc8104da0 2024-03-24T10:34:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc819cfd0 con 0x7fdbc8104da0 2024-03-24T10:34:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 3) v1 ==== 45431+0+0 (secure 0 0 0) 0x7fdbc401b440 con 0x7fdbc8104da0 2024-03-24T10:34:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.201+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc8196120 con 0x7fdbc8104da0 2024-03-24T10:34:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.202+0000 7fdbbeffd700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 0x7fdbb403aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.202+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdbc404c6b0 con 0x7fdbc8104da0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.202+0000 7fdbcd618700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 0x7fdbb403aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.202+0000 7fdbcd618700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 0x7fdbb403aac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8006fd0 tx=0x7fdbb8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.207+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdbc40199d0 con 0x7fdbc8104da0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.252+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7fdbc4019d50 con 0x7fdbc8104da0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.491+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdbc802ccb0 con 0x7fdbc8104da0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.492+0000 7fdbbeffd700 1 -- 172.21.15.26:0/2400404149 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fdbc401f4f0 con 0x7fdbc8104da0 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 msgr2=0x7fdbb403aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 0x7fdbb403aac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8006fd0 tx=0x7fdbb8006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 msgr2=0x7fdbc819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4011710 tx=0x7fdbc40047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 shutdown_connections 2024-03-24T10:34:21.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fdbb4038600 0x7fdbb403aac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 --2- 172.21.15.26:0/2400404149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbc8104da0 0x7fdbc819c3f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 >> 172.21.15.26:0/2400404149 conn(0x7fdbc81009c0 msgr2=0x7fdbc8193260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 shutdown_connections 2024-03-24T10:34:21.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:20.494+0000 7fdbd007d700 1 -- 172.21.15.26:0/2400404149 wait complete. 2024-03-24T10:34:21.118 INFO:teuthology.orchestra.run.smithi026.stdout:mgr is available 2024-03-24T10:34:21.351 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:21 smithi026 ceph-mon[29210]: mgrmap e4: smithi026.bbjcru(active, since 3s) 2024-03-24T10:34:21.351 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:21 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2400404149' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T10:34:22.311 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.543+0000 7fe5377a4700 1 Processor -- start 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.543+0000 7fe5377a4700 1 -- start start 2024-03-24T10:34:22.312 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.543+0000 7fe5377a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.543+0000 7fe5377a4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300745f0 con 0x7fe530106f30 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe535540700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe535540700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50402/0 (socket says 172.21.15.26:50402) 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe535540700 1 -- 172.21.15.26:0/865396744 learned_addr learned my addr 172.21.15.26:0/865396744 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe535540700 1 -- 172.21.15.26:0/865396744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530109860 con 0x7fe530106f30 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe535540700 1 --2- 172.21.15.26:0/865396744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009e80 tx=0x7fe52c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cb1efd9d947f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe527fff700 1 -- 172.21.15.26:0/865396744 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe52c00e070 con 0x7fe530106f30 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe527fff700 1 -- 172.21.15.26:0/865396744 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe52c00b8c0 con 0x7fe530106f30 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.544+0000 7fe527fff700 1 -- 172.21.15.26:0/865396744 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe52c004440 con 0x7fe530106f30 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 -- 172.21.15.26:0/865396744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 msgr2=0x7fe530109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 --2- 172.21.15.26:0/865396744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009e80 tx=0x7fe52c00b330 comp rx=0 tx=0).stop 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 -- 172.21.15.26:0/865396744 shutdown_connections 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 --2- 172.21.15.26:0/865396744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe530109320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.313 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 -- 172.21.15.26:0/865396744 >> 172.21.15.26:0/865396744 conn(0x7fe5301009c0 msgr2=0x7fe530102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 -- 172.21.15.26:0/865396744 shutdown_connections 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.545+0000 7fe5377a4700 1 -- 172.21.15.26:0/865396744 wait complete. 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5377a4700 1 Processor -- start 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5377a4700 1 -- start start 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5377a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5377a4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53019a6d0 con 0x7fe530106f30 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe535540700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe535540700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50404/0 (socket says 172.21.15.26:50404) 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe535540700 1 -- 172.21.15.26:0/1656700095 learned_addr learned my addr 172.21.15.26:0/1656700095 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe535540700 1 -- 172.21.15.26:0/1656700095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c009b30 con 0x7fe530106f30 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe535540700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003c50 tx=0x7fe52c003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe52c00e040 con 0x7fe530106f30 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53019a8d0 con 0x7fe530106f30 2024-03-24T10:34:22.314 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.546+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe52c00bc40 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.547+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe52c0125e0 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.547+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53019add0 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.547+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7fe52c019040 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.548+0000 7fe5267fc700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 0x7fe51c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.548+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53010a5e0 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.548+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe52c04bd00 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.548+0000 7fe534d3f700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 0x7fe51c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.548+0000 7fe534d3f700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 0x7fe51c03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe520006fd0 tx=0x7fe520006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.552+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe52c01f080 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.752+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe53019dae0 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.753+0000 7fe5267fc700 1 -- 172.21.15.26:0/1656700095 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe52c01a570 con 0x7fe530106f30 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 msgr2=0x7fe51c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 0x7fe51c03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe520006fd0 tx=0x7fe520006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:22.315 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 msgr2=0x7fe53019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003c50 tx=0x7fe52c003d30 comp rx=0 tx=0).stop 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 shutdown_connections 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe51c0386b0 0x7fe51c03ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 --2- 172.21.15.26:0/1656700095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe530106f30 0x7fe53019a190 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 >> 172.21.15.26:0/1656700095 conn(0x7fe5301009c0 msgr2=0x7fe530101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 shutdown_connections 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:21.755+0000 7fe5377a4700 1 -- 172.21.15.26:0/1656700095 wait complete. 2024-03-24T10:34:22.316 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling cephadm module... 2024-03-24T10:34:22.541 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:22 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1656700095' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T10:34:23.273 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:23 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1312481723' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.753+0000 7fe4faf5e700 1 Processor -- start 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.753+0000 7fe4faf5e700 1 -- start start 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.753+0000 7fe4faf5e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4faf5e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4105db0 con 0x7fe4f41053c0 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f8cfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f8cfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44416/0 (socket says 172.21.15.26:44416) 2024-03-24T10:34:23.841 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f8cfa700 1 -- 172.21.15.26:0/914771442 learned_addr learned my addr 172.21.15.26:0/914771442 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:23.841 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f8cfa700 1 -- 172.21.15.26:0/914771442 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f41065c0 con 0x7fe4f41053c0 2024-03-24T10:34:23.841 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f8cfa700 1 --2- 172.21.15.26:0/914771442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80098a0 tx=0x7fe4e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464e9bf08c8673bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.841 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f37fe700 1 -- 172.21.15.26:0/914771442 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e8004180 con 0x7fe4f41053c0 2024-03-24T10:34:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f37fe700 1 -- 172.21.15.26:0/914771442 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe4e80042e0 con 0x7fe4f41053c0 2024-03-24T10:34:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.754+0000 7fe4f37fe700 1 -- 172.21.15.26:0/914771442 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e8011a70 con 0x7fe4f41053c0 2024-03-24T10:34:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 -- 172.21.15.26:0/914771442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 msgr2=0x7fe4f41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/914771442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80098a0 tx=0x7fe4e8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 -- 172.21.15.26:0/914771442 shutdown_connections 2024-03-24T10:34:23.844 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/914771442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f41057e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.844 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 -- 172.21.15.26:0/914771442 >> 172.21.15.26:0/914771442 conn(0x7fe4f41009c0 msgr2=0x7fe4f4102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.846 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 -- 172.21.15.26:0/914771442 shutdown_connections 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.755+0000 7fe4faf5e700 1 -- 172.21.15.26:0/914771442 wait complete. 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4faf5e700 1 Processor -- start 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4faf5e700 1 -- start start 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4faf5e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4faf5e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4074680 con 0x7fe4f41053c0 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4f8cfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.847 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4f8cfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44420/0 (socket says 172.21.15.26:44420) 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4f8cfa700 1 -- 172.21.15.26:0/1312481723 learned_addr learned my addr 172.21.15.26:0/1312481723 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4f8cfa700 1 -- 172.21.15.26:0/1312481723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8009550 con 0x7fe4f41053c0 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.756+0000 7fe4f8cfa700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8011710 tx=0x7fe4e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.757+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e8003b10 con 0x7fe4f41053c0 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.757+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe4e801b8c0 con 0x7fe4f41053c0 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.757+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e8019770 con 0x7fe4f41053c0 2024-03-24T10:34:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.757+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f419cf60 con 0x7fe4f41053c0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.757+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f419d3a0 con 0x7fe4f41053c0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7fe4e801ba30 con 0x7fe4f41053c0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f41097f0 con 0x7fe4f41053c0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4f1ffb700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 0x7fe4e403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe4e804c860 con 0x7fe4f41053c0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4f3fff700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 0x7fe4e403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.849 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.758+0000 7fe4f3fff700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 0x7fe4e403abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006fd0 tx=0x7fe4e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:22.763+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe4e8018350 con 0x7fe4f41053c0 2024-03-24T10:34:23.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.020+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe4f419da00 con 0x7fe4f41053c0 2024-03-24T10:34:23.850 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.257+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fe4e801b440 con 0x7fe4f41053c0 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.257+0000 7fe4f1ffb700 1 -- 172.21.15.26:0/1312481723 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe4e804c280 con 0x7fe4f41053c0 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.260+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 msgr2=0x7fe4e403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.260+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 0x7fe4e403abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006fd0 tx=0x7fe4e0006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.260+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 msgr2=0x7fe4f4074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8011710 tx=0x7fe4e80047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.851 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 shutdown_connections 2024-03-24T10:34:23.852 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7fe4e4038720 0x7fe4e403abe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.852 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 --2- 172.21.15.26:0/1312481723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4f41053c0 0x7fe4f4074140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.852 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 >> 172.21.15.26:0/1312481723 conn(0x7fe4f41009c0 msgr2=0x7fe4f4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.852 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 shutdown_connections 2024-03-24T10:34:23.852 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:23.261+0000 7fe4faf5e700 1 -- 172.21.15.26:0/1312481723 wait complete. 2024-03-24T10:34:24.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:24 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1312481723' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T10:34:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:24 smithi026 ceph-mon[29210]: mgrmap e5: smithi026.bbjcru(active, since 6s) 2024-03-24T10:34:25.173 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:25.173 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "smithi026.bbjcru", 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5eb40700 1 Processor -- start 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5eb40700 1 -- start start 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5eb40700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5eb40700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b58105db0 con 0x7f3b581053c0 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5c8dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5c8dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44450/0 (socket says 172.21.15.26:44450) 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.251+0000 7f3b5c8dc700 1 -- 172.21.15.26:0/3915869446 learned_addr learned my addr 172.21.15.26:0/3915869446 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.252+0000 7f3b5c8dc700 1 -- 172.21.15.26:0/3915869446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b581065c0 con 0x7f3b581053c0 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.252+0000 7f3b5c8dc700 1 --2- 172.21.15.26:0/3915869446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0098a0 tx=0x7f3b4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece6db179cdc206e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.252+0000 7f3b577fe700 1 -- 172.21.15.26:0/3915869446 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c004180 con 0x7f3b581053c0 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.252+0000 7f3b577fe700 1 -- 172.21.15.26:0/3915869446 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3b4c0042e0 con 0x7f3b581053c0 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.252+0000 7f3b577fe700 1 -- 172.21.15.26:0/3915869446 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c011a70 con 0x7f3b581053c0 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- 172.21.15.26:0/3915869446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 msgr2=0x7f3b581057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.175 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/3915869446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0098a0 tx=0x7f3b4c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- 172.21.15.26:0/3915869446 shutdown_connections 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/3915869446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b581057e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- 172.21.15.26:0/3915869446 >> 172.21.15.26:0/3915869446 conn(0x7f3b581009c0 msgr2=0x7f3b58102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- 172.21.15.26:0/3915869446 shutdown_connections 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- 172.21.15.26:0/3915869446 wait complete. 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 Processor -- start 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.253+0000 7f3b5eb40700 1 -- start start 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5eb40700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5eb40700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b581909a0 con 0x7f3b581053c0 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5c8dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5c8dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44454/0 (socket says 172.21.15.26:44454) 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5c8dc700 1 -- 172.21.15.26:0/459217580 learned_addr learned my addr 172.21.15.26:0/459217580 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5c8dc700 1 -- 172.21.15.26:0/459217580 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c009550 con 0x7f3b581053c0 2024-03-24T10:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5c8dc700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c011710 tx=0x7f3b4c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c003b10 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3b4c01b8c0 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5818ebe0 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.254+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4c019770 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.255+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5818f080 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.255+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f3b4c0199a0 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.256+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b580623e0 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.256+0000 7f3b55ffb700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 0x7f3b4803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.256+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3b4c04c8b0 con 0x7f3b581053c0 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.256+0000 7f3b57fff700 1 -- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 msgr2=0x7f3b4803ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.256+0000 7f3b57fff700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 0x7f3b4803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:34:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.263+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b4c018db0 con 0x7f3b581053c0 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.457+0000 7f3b57fff700 1 -- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 msgr2=0x7f3b4803ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.457+0000 7f3b57fff700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 0x7f3b4803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.523+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3b581943c0 con 0x7f3b581053c0 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.524+0000 7f3b55ffb700 1 -- 172.21.15.26:0/459217580 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3b4c01f070 con 0x7f3b581053c0 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 msgr2=0x7f3b4803ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 0x7f3b4803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 msgr2=0x7f3b58190460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c011710 tx=0x7f3b4c0047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 shutdown_connections 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f3b48038770 0x7f3b4803ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 --2- 172.21.15.26:0/459217580 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b581053c0 0x7f3b58190460 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 >> 172.21.15.26:0/459217580 conn(0x7f3b581009c0 msgr2=0x7f3b58101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 shutdown_connections 2024-03-24T10:34:25.178 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:24.526+0000 7f3b5eb40700 1 -- 172.21.15.26:0/459217580 wait complete. 2024-03-24T10:34:25.179 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-24T10:34:25.179 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 5... 2024-03-24T10:34:25.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:25 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/459217580' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T10:34:31.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: Active manager daemon smithi026.bbjcru restarted 2024-03-24T10:34:31.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: Activating manager daemon smithi026.bbjcru 2024-03-24T10:34:31.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: mgrmap e6: smithi026.bbjcru(active, starting, since 0.00323748s) 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi026.bbjcru", "id": "smithi026.bbjcru"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: Manager daemon smithi026.bbjcru is now available 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: Found migration_current of "None". Setting to last migration. 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:31.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:34:31.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:31 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/trash_purge_schedule"}]: dispatch 2024-03-24T10:34:31.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:31.804 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d08d12700 1 Processor -- start 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d08d12700 1 -- start start 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d08d12700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d08d12700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04105db0 con 0x7f9d041053c0 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44458/0 (socket says 172.21.15.26:44458) 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d0259c700 1 -- 172.21.15.26:0/1223075650 learned_addr learned my addr 172.21.15.26:0/1223075650 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.600+0000 7f9d0259c700 1 -- 172.21.15.26:0/1223075650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d041065c0 con 0x7f9d041053c0 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d0259c700 1 --2- 172.21.15.26:0/1223075650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40098a0 tx=0x7f9cf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b7121b1f326920 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d0159a700 1 -- 172.21.15.26:0/1223075650 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4004180 con 0x7f9d041053c0 2024-03-24T10:34:31.805 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d0159a700 1 -- 172.21.15.26:0/1223075650 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9cf40042e0 con 0x7f9d041053c0 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d0159a700 1 -- 172.21.15.26:0/1223075650 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4011a70 con 0x7f9d041053c0 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d08d12700 1 -- 172.21.15.26:0/1223075650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 msgr2=0x7f9d041057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.601+0000 7f9d08d12700 1 --2- 172.21.15.26:0/1223075650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40098a0 tx=0x7f9cf4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- 172.21.15.26:0/1223075650 shutdown_connections 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 --2- 172.21.15.26:0/1223075650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d041057e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- 172.21.15.26:0/1223075650 >> 172.21.15.26:0/1223075650 conn(0x7f9d041009c0 msgr2=0x7f9d04102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- 172.21.15.26:0/1223075650 shutdown_connections 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- 172.21.15.26:0/1223075650 wait complete. 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 Processor -- start 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- start start 2024-03-24T10:34:31.806 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.602+0000 7f9d08d12700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04074680 con 0x7f9d041053c0 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44462/0 (socket says 172.21.15.26:44462) 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d0259c700 1 -- 172.21.15.26:0/163077874 learned_addr learned my addr 172.21.15.26:0/163077874 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d0259c700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4009550 con 0x7f9d041053c0 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d0259c700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011710 tx=0x7f9cf40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4003b10 con 0x7f9d041053c0 2024-03-24T10:34:31.807 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9cf401b8c0 con 0x7f9d041053c0 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0419cf60 con 0x7f9d041053c0 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4019770 con 0x7f9d041053c0 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.603+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0419d3a0 con 0x7f9d041053c0 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f9cf40199a0 con 0x7f9d041053c0 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9cf37fe700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cec03b340 con 0x7f9cec038770 2024-03-24T10:34:31.808 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cf404c840 con 0x7f9d041053c0 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9d01d9b700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.604+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.805+0000 7f9d01d9b700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:25.805+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:26.206+0000 7f9d01d9b700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:26.206+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:27.007+0000 7f9d01d9b700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:27.007+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:28.608+0000 7f9d01d9b700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/4217413357 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:28.608+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:30.201+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 6) v1 ==== 45315+0+0 (secure 0 0 0) 0x7f9cf400ed00 con 0x7f9d041053c0 2024-03-24T10:34:31.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:30.202+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 msgr2=0x7f9cec03ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:30.202+0000 7f9cf37fe700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.202+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7f9cf404e150 con 0x7f9d041053c0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.203+0000 7f9cf37fe700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 0x7f9cec03e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.203+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cec03b340 con 0x7f9cec03bfb0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.203+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 0x7f9cec03e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.204+0000 7f9d01d9b700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 0x7f9cec03e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8003a10 tx=0x7f9cf80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.204+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f9cec03b340 con 0x7f9cec03bfb0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9d040623e0 con 0x7f9cec03bfb0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9cf37fe700 1 -- 172.21.15.26:0/163077874 <== mgr.14120 v2:172.21.15.26:6800/3234553684 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9d040623e0 con 0x7f9cec03bfb0 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 msgr2=0x7f9cec03e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 0x7f9cec03e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8003a10 tx=0x7f9cf80092b0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 msgr2=0x7f9d04074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011710 tx=0x7f9cf40047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 shutdown_connections 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9cec03bfb0 0x7f9cec03e3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d041053c0 0x7f9d04074140 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 --2- 172.21.15.26:0/163077874 >> [v2:172.21.15.26:6800/4217413357,v1:172.21.15.26:6801/4217413357] conn(0x7f9cec038770 0x7f9cec03ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 >> 172.21.15.26:0/163077874 conn(0x7f9d041009c0 msgr2=0x7f9d04101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 shutdown_connections 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:31.211+0000 7f9d08d12700 1 -- 172.21.15.26:0/163077874 wait complete. 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 5 is available 2024-03-24T10:34:31.812 INFO:teuthology.orchestra.run.smithi026.stdout:Setting orchestrator backend to cephadm... 2024-03-24T10:34:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:32 smithi026 ceph-mon[29210]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T10:34:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:32 smithi026 ceph-mon[29210]: mgrmap e7: smithi026.bbjcru(active, since 1.00518s) 2024-03-24T10:34:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:32 smithi026 ceph-mon[29210]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T10:34:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:32 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:32 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f224d421700 1 Processor -- start 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f224d421700 1 -- start start 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f224d421700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f224d421700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480745b0 con 0x7f2248106f70 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44512/0 (socket says 172.21.15.26:44512) 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.239+0000 7f2246ffd700 1 -- 172.21.15.26:0/2288246081 learned_addr learned my addr 172.21.15.26:0/2288246081 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.240+0000 7f2246ffd700 1 -- 172.21.15.26:0/2288246081 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22481098a0 con 0x7f2248106f70 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.240+0000 7f2246ffd700 1 --2- 172.21.15.26:0/2288246081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f22380098a0 tx=0x7f2238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89966e34bfe31159 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.240+0000 7f2245ffb700 1 -- 172.21.15.26:0/2288246081 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2238004180 con 0x7f2248106f70 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.240+0000 7f2245ffb700 1 -- 172.21.15.26:0/2288246081 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f22380042e0 con 0x7f2248106f70 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.240+0000 7f2245ffb700 1 -- 172.21.15.26:0/2288246081 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2238011a70 con 0x7f2248106f70 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- 172.21.15.26:0/2288246081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 msgr2=0x7f2248109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 --2- 172.21.15.26:0/2288246081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f22380098a0 tx=0x7f2238009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- 172.21.15.26:0/2288246081 shutdown_connections 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 --2- 172.21.15.26:0/2288246081 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f2248109360 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- 172.21.15.26:0/2288246081 >> 172.21.15.26:0/2288246081 conn(0x7f22481009c0 msgr2=0x7f2248102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- 172.21.15.26:0/2288246081 shutdown_connections 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- 172.21.15.26:0/2288246081 wait complete. 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 Processor -- start 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.241+0000 7f224d421700 1 -- start start 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f224d421700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f224d421700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224819a780 con 0x7f2248106f70 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44514/0 (socket says 172.21.15.26:44514) 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f2246ffd700 1 -- 172.21.15.26:0/1363333180 learned_addr learned my addr 172.21.15.26:0/1363333180 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f2246ffd700 1 -- 172.21.15.26:0/1363333180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009550 con 0x7f2248106f70 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f2246ffd700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2238011710 tx=0x7f22380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2238003b10 con 0x7f2248106f70 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224819a980 con 0x7f2248106f70 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f223801b8c0 con 0x7f2248106f70 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22380197e0 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.242+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224819ae20 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.243+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7f2238019a20 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.243+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224810a5f0 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.243+0000 7f223ffff700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 0x7f223403aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.243+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f223804c940 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.243+0000 7f22467fc700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 0x7f223403aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.244+0000 7f22467fc700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 0x7f223403aac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.248+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2238018350 con 0x7f2248106f70 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.465+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2248101bc0 con 0x7f2234038600 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.474+0000 7f223ffff700 1 -- 172.21.15.26:0/1363333180 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2248101bc0 con 0x7f2234038600 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 msgr2=0x7f223403aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 0x7f223403aac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 msgr2=0x7f224819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2238011710 tx=0x7f22380047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 shutdown_connections 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2234038600 0x7f223403aac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 --2- 172.21.15.26:0/1363333180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2248106f70 0x7f224819a240 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.476+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 >> 172.21.15.26:0/1363333180 conn(0x7f22481009c0 msgr2=0x7f22481014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.477+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 shutdown_connections 2024-03-24T10:34:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:32.477+0000 7f224d421700 1 -- 172.21.15.26:0/1363333180 wait complete. 2024-03-24T10:34:33.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:33.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:33.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: [24/Mar/2024:10:34:32] ENGINE Bus STARTING 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: [24/Mar/2024:10:34:33] ENGINE Serving on http://172.21.15.26:8765 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: mgrmap e8: smithi026.bbjcru(active, since 3s) 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: [24/Mar/2024:10:34:33] ENGINE Serving on https://172.21.15.26:7150 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: [24/Mar/2024:10:34:33] ENGINE Bus STARTED 2024-03-24T10:34:33.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:33 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.453+0000 7f9667743700 1 Processor -- start 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.453+0000 7f9667743700 1 -- start start 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.453+0000 7f9667743700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.453+0000 7f9667743700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660106f20 con 0x7f9660106530 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f96654df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f96654df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35678/0 (socket says 172.21.15.26:35678) 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f96654df700 1 -- 172.21.15.26:0/1420679173 learned_addr learned my addr 172.21.15.26:0/1420679173 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f96654df700 1 -- 172.21.15.26:0/1420679173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660107690 con 0x7f9660106530 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f96654df700 1 --2- 172.21.15.26:0/1420679173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9650009e80 tx=0x7f965000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce841a164b8c7254 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f965ffff700 1 -- 172.21.15.26:0/1420679173 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965000e070 con 0x7f9660106530 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f965ffff700 1 -- 172.21.15.26:0/1420679173 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f965000b8c0 con 0x7f9660106530 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.454+0000 7f965ffff700 1 -- 172.21.15.26:0/1420679173 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9650004440 con 0x7f9660106530 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- 172.21.15.26:0/1420679173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 msgr2=0x7f9660106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 --2- 172.21.15.26:0/1420679173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9650009e80 tx=0x7f965000b330 comp rx=0 tx=0).stop 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- 172.21.15.26:0/1420679173 shutdown_connections 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 --2- 172.21.15.26:0/1420679173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f9660106950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- 172.21.15.26:0/1420679173 >> 172.21.15.26:0/1420679173 conn(0x7f9660101b10 msgr2=0x7f9660103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- 172.21.15.26:0/1420679173 shutdown_connections 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- 172.21.15.26:0/1420679173 wait complete. 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 Processor -- start 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.455+0000 7f9667743700 1 -- start start 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f9667743700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f9667743700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966019c920 con 0x7f9660106530 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f96654df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f96654df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35680/0 (socket says 172.21.15.26:35680) 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f96654df700 1 -- 172.21.15.26:0/4211647282 learned_addr learned my addr 172.21.15.26:0/4211647282 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f96654df700 1 -- 172.21.15.26:0/4211647282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650009b30 con 0x7f9660106530 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f96654df700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f96500099e0 tx=0x7f9650003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965000e040 con 0x7f9660106530 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f965000bc40 con 0x7f9660106530 2024-03-24T10:34:34.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966019cb20 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96500125e0 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.456+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966019d020 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.457+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f9650019040 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.457+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96600623e0 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.457+0000 7f965e7fc700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 0x7f964803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.457+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f965004c040 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.457+0000 7f9664cde700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 0x7f964803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.458+0000 7f9664cde700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 0x7f964803abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9654006fd0 tx=0x7f9654006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.462+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f965001a570 con 0x7f9660106530 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.667+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9660102cf0 con 0x7f9648038720 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.667+0000 7f965e7fc700 1 -- 172.21.15.26:0/4211647282 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9660102cf0 con 0x7f9648038720 2024-03-24T10:34:34.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 msgr2=0x7f964803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 0x7f964803abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9654006fd0 tx=0x7f9654006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 msgr2=0x7f966019c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f96500099e0 tx=0x7f9650003d00 comp rx=0 tx=0).stop 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 shutdown_connections 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f9648038720 0x7f964803abe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 --2- 172.21.15.26:0/4211647282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9660106530 0x7f966019c3e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.669+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 >> 172.21.15.26:0/4211647282 conn(0x7f9660101b10 msgr2=0x7f96601025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.670+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 shutdown_connections 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:33.670+0000 7f9667743700 1 -- 172.21.15.26:0/4211647282 wait complete. 2024-03-24T10:34:34.204 INFO:teuthology.orchestra.run.smithi026.stdout:Generating ssh key... 2024-03-24T10:34:34.610 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:34 smithi026 ceph-mon[29210]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91fda8c700 1 Processor -- start 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91fda8c700 1 -- start start 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91fda8c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91fda8c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8105df0 con 0x7f91f8105400 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35688/0 (socket says 172.21.15.26:35688) 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91f77fe700 1 -- 172.21.15.26:0/1387613193 learned_addr learned my addr 172.21.15.26:0/1387613193 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:35.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.636+0000 7f91f77fe700 1 -- 172.21.15.26:0/1387613193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f8106600 con 0x7f91f8105400 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91f77fe700 1 --2- 172.21.15.26:0/1387613193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098a0 tx=0x7f91e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0009015a9e86f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91f67fc700 1 -- 172.21.15.26:0/1387613193 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8004180 con 0x7f91f8105400 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91f67fc700 1 -- 172.21.15.26:0/1387613193 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f91e80042e0 con 0x7f91f8105400 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91f67fc700 1 -- 172.21.15.26:0/1387613193 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8011a70 con 0x7f91f8105400 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91fda8c700 1 -- 172.21.15.26:0/1387613193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 msgr2=0x7f91f8105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91fda8c700 1 --2- 172.21.15.26:0/1387613193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098a0 tx=0x7f91e8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91fda8c700 1 -- 172.21.15.26:0/1387613193 shutdown_connections 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91fda8c700 1 --2- 172.21.15.26:0/1387613193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f8105820 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.637+0000 7f91fda8c700 1 -- 172.21.15.26:0/1387613193 >> 172.21.15.26:0/1387613193 conn(0x7f91f8100a00 msgr2=0x7f91f8102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.638+0000 7f91fda8c700 1 -- 172.21.15.26:0/1387613193 shutdown_connections 2024-03-24T10:34:35.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.638+0000 7f91fda8c700 1 -- 172.21.15.26:0/1387613193 wait complete. 2024-03-24T10:34:35.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.638+0000 7f91fda8c700 1 Processor -- start 2024-03-24T10:34:35.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.638+0000 7f91fda8c700 1 -- start start 2024-03-24T10:34:35.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91fda8c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91fda8c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f819a790 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35696/0 (socket says 172.21.15.26:35696) 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f77fe700 1 -- 172.21.15.26:0/503983843 learned_addr learned my addr 172.21.15.26:0/503983843 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f77fe700 1 -- 172.21.15.26:0/503983843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8009550 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f77fe700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011710 tx=0x7f91e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8003b10 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f91e801b8c0 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8019770 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f819a990 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.639+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f819ae30 con 0x7f91f8105400 2024-03-24T10:34:35.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.640+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f91e80199a0 con 0x7f91f8105400 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.640+0000 7f91f4ff9700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 0x7f91e003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.640+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f8194420 con 0x7f91f8105400 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.640+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f91e804cb60 con 0x7f91f8105400 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.641+0000 7f91f6ffd700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 0x7f91e003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.641+0000 7f91f6ffd700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 0x7f91e003ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006fd0 tx=0x7f91ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.645+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f91e801a3d0 con 0x7f91f8105400 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:34.848+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f91f80611f0 con 0x7f91e0038770 2024-03-24T10:34:35.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.197+0000 7f91f4ff9700 1 -- 172.21.15.26:0/503983843 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f91f80611f0 con 0x7f91e0038770 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 msgr2=0x7f91e003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 0x7f91e003ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006fd0 tx=0x7f91ec006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 msgr2=0x7f91f819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011710 tx=0x7f91e80047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 shutdown_connections 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f91e0038770 0x7f91e003ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 --2- 172.21.15.26:0/503983843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91f8105400 0x7f91f819a250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 >> 172.21.15.26:0/503983843 conn(0x7f91f8100a00 msgr2=0x7f91f8101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 shutdown_connections 2024-03-24T10:34:35.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:35.199+0000 7f91fda8c700 1 -- 172.21.15.26:0/503983843 wait complete. 2024-03-24T10:34:36.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:36 smithi026 ceph-mon[29210]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:36.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:36 smithi026 ceph-mon[29210]: Generating ssh key... 2024-03-24T10:34:36.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:36 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:36.414 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:36 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:36.987 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdaXINS/3gAlWcLH0Wu/0967EWjrL4Kmf1YI7HEkOHAVCU23OdjS45cDY/CaPFGSyYcqZRUXGUSJsPBC+3mjBoeDNIacj15vsdiY5UCYbZahOZyPqOilkK0N5qMXiW6BO0G4NLKQvP6mmzUeDT+PkfBhwKyDcJLs0WwXRSYsWmdxHcdT9ZOFnRL3tNv5J8/eA4SmbLhXEOltw+YOxWVinwFMp4V4s56Aet8Ggu2OrAXopMoTFK4GdQ81Hm665PHeXWTwIOQa8x1PIMvrpLMXyr1hkBt9IPrWEb+OP+SEDJyjdTMbQBOTKxFr5OjLSiJMOOgS4JNWboCizptOUhQB7dJwQRBEg8EnVfRRwLi63ZZr5DVvujdvJZGbx3s0cwvFZhLM0zPnmOpPfC84TJHbGyJEtBst61Rlnw34P4BcYxgrRBqnjILkJ9222g1eM5e30E5ZXwdw+qHmx3QJgGElVPLVnXoyuEJXbkVfcH3b4FUw3K1qHk9aa5wtknglm6Exs= ceph-e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:34:36.987 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.199+0000 7f2a9c44a700 1 Processor -- start 2024-03-24T10:34:36.987 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.199+0000 7f2a9c44a700 1 -- start start 2024-03-24T10:34:36.987 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.199+0000 7f2a9c44a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.199+0000 7f2a9c44a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94079db0 con 0x7f2a94107010 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a9a1e6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a9a1e6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35708/0 (socket says 172.21.15.26:35708) 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a9a1e6700 1 -- 172.21.15.26:0/1481819503 learned_addr learned my addr 172.21.15.26:0/1481819503 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a9a1e6700 1 -- 172.21.15.26:0/1481819503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94079ef0 con 0x7f2a94107010 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a9a1e6700 1 --2- 172.21.15.26:0/1481819503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a900098a0 tx=0x7f2a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f1a98acda03b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a991e4700 1 -- 172.21.15.26:0/1481819503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a90004180 con 0x7f2a94107010 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a991e4700 1 -- 172.21.15.26:0/1481819503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2a900042e0 con 0x7f2a94107010 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.200+0000 7f2a991e4700 1 -- 172.21.15.26:0/1481819503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a90011a70 con 0x7f2a94107010 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1481819503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 msgr2=0x7f2a94109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1481819503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a900098a0 tx=0x7f2a90009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1481819503 shutdown_connections 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1481819503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94109400 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.988 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1481819503 >> 172.21.15.26:0/1481819503 conn(0x7f2a94100980 msgr2=0x7f2a94102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1481819503 shutdown_connections 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.201+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1481819503 wait complete. 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9c44a700 1 Processor -- start 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9c44a700 1 -- start start 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9c44a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9c44a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94106d80 con 0x7f2a94107010 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9a1e6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9a1e6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35722/0 (socket says 172.21.15.26:35722) 2024-03-24T10:34:36.989 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9a1e6700 1 -- 172.21.15.26:0/1524631263 learned_addr learned my addr 172.21.15.26:0/1524631263 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.202+0000 7f2a9a1e6700 1 -- 172.21.15.26:0/1524631263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90009550 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a9a1e6700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011710 tx=0x7f2a900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a90003b10 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2a9001b8c0 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94104fc0 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a900197e0 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.203+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94105460 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f2a90019a20 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9410a5f0 con 0x7f2a94107010 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a8b7fe700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 0x7f2a8003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.990 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2a9004cb60 con 0x7f2a94107010 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a999e5700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 0x7f2a8003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.204+0000 7f2a999e5700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 0x7f2a8003abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.209+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2a90018db0 con 0x7f2a94107010 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.413+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a94101c00 con 0x7f2a80038720 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.414+0000 7f2a8b7fe700 1 -- 172.21.15.26:0/1524631263 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2a94101c00 con 0x7f2a80038720 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 msgr2=0x7f2a8003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 0x7f2a8003abe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 msgr2=0x7f2a94106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011710 tx=0x7f2a900047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 shutdown_connections 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2a80038720 0x7f2a8003abe0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 --2- 172.21.15.26:0/1524631263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a94107010 0x7f2a94106840 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 >> 172.21.15.26:0/1524631263 conn(0x7f2a94100980 msgr2=0x7f2a941014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.991 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 shutdown_connections 2024-03-24T10:34:36.992 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:36.416+0000 7f2a9c44a700 1 -- 172.21.15.26:0/1524631263 wait complete. 2024-03-24T10:34:36.992 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:34:36.992 INFO:teuthology.orchestra.run.smithi026.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T10:34:36.992 INFO:teuthology.orchestra.run.smithi026.stdout:Adding host smithi026... 2024-03-24T10:34:37.647 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:37 smithi026 ceph-mon[29210]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:38.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:38 smithi026 ceph-mon[29210]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:39.695 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:39 smithi026 ceph-mon[29210]: Deploying cephadm binary to smithi026 2024-03-24T10:34:40.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Added host 'smithi026' with addr '172.21.15.26' 2024-03-24T10:34:40.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.401+0000 7ff5037ad700 1 Processor -- start 2024-03-24T10:34:40.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.401+0000 7ff5037ad700 1 -- start start 2024-03-24T10:34:40.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.401+0000 7ff5037ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.401+0000 7ff5037ad700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc106ee0 con 0x7ff4fc1064f0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff501549700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff501549700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35728/0 (socket says 172.21.15.26:35728) 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff501549700 1 -- 172.21.15.26:0/1980428755 learned_addr learned my addr 172.21.15.26:0/1980428755 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff501549700 1 -- 172.21.15.26:0/1980428755 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc1076a0 con 0x7ff4fc1064f0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff501549700 1 --2- 172.21.15.26:0/1980428755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40098a0 tx=0x7ff4f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e050208c57d4c4b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff4fbfff700 1 -- 172.21.15.26:0/1980428755 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4004180 con 0x7ff4fc1064f0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff4fbfff700 1 -- 172.21.15.26:0/1980428755 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff4f40042e0 con 0x7ff4fc1064f0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.402+0000 7ff4fbfff700 1 -- 172.21.15.26:0/1980428755 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4011a70 con 0x7ff4fc1064f0 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 -- 172.21.15.26:0/1980428755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 msgr2=0x7ff4fc106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 --2- 172.21.15.26:0/1980428755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40098a0 tx=0x7ff4f4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 -- 172.21.15.26:0/1980428755 shutdown_connections 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 --2- 172.21.15.26:0/1980428755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc106910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 -- 172.21.15.26:0/1980428755 >> 172.21.15.26:0/1980428755 conn(0x7ff4fc101b70 msgr2=0x7ff4fc103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 -- 172.21.15.26:0/1980428755 shutdown_connections 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.403+0000 7ff5037ad700 1 -- 172.21.15.26:0/1980428755 wait complete. 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff5037ad700 1 Processor -- start 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff5037ad700 1 -- start start 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff5037ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff5037ad700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc07c040 con 0x7ff4fc1064f0 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff501549700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff501549700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35736/0 (socket says 172.21.15.26:35736) 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff501549700 1 -- 172.21.15.26:0/465673824 learned_addr learned my addr 172.21.15.26:0/465673824 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff501549700 1 -- 172.21.15.26:0/465673824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f4009550 con 0x7ff4fc1064f0 2024-03-24T10:34:40.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff501549700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4011710 tx=0x7ff4f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.404+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4003b10 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.405+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4fc07a280 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.405+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4fc07a720 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.405+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff4f401b8c0 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.405+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f4019710 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.405+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7ff4f4019940 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.406+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e8005320 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.406+0000 7ff4fa7fc700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 0x7ff4e403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.406+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff4f404ca70 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.406+0000 7ff500d48700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 0x7ff4e403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.406+0000 7ff500d48700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 0x7ff4e403abf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006fd0 tx=0x7ff4f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.411+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff4f4018b80 con 0x7ff4fc1064f0 2024-03-24T10:34:40.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:37.646+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4e8000bf0 con 0x7ff4e4038730 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.313+0000 7ff4fa7fc700 1 -- 172.21.15.26:0/465673824 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7ff4e8000bf0 con 0x7ff4e4038730 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 msgr2=0x7ff4e403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 0x7ff4e403abf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006fd0 tx=0x7ff4f0006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 msgr2=0x7ff4fc07bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4011710 tx=0x7ff4f40047c0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 shutdown_connections 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7ff4e4038730 0x7ff4e403abf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 --2- 172.21.15.26:0/465673824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4fc1064f0 0x7ff4fc07bb00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 >> 172.21.15.26:0/465673824 conn(0x7ff4fc101b70 msgr2=0x7ff4fc102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 shutdown_connections 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:40.316+0000 7ff5037ad700 1 -- 172.21.15.26:0/465673824 wait complete. 2024-03-24T10:34:40.916 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying mon service with default placement... 2024-03-24T10:34:41.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:41 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:41.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:41 smithi026 ceph-mon[29210]: Added host smithi026 2024-03-24T10:34:41.533 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:41 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:42.925 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:42 smithi026 ceph-mon[29210]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:42.925 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:42 smithi026 ceph-mon[29210]: Saving service mon spec with placement count:5 2024-03-24T10:34:42.925 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:42 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:43.117 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.338+0000 7f472f95f700 1 Processor -- start 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.338+0000 7f472f95f700 1 -- start start 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.338+0000 7f472f95f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.338+0000 7f472f95f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47280745f0 con 0x7f4728106fc0 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f472d6fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f472d6fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35752/0 (socket says 172.21.15.26:35752) 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f472d6fb700 1 -- 172.21.15.26:0/1449022350 learned_addr learned my addr 172.21.15.26:0/1449022350 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f472d6fb700 1 -- 172.21.15.26:0/1449022350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47281098f0 con 0x7f4728106fc0 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f472d6fb700 1 --2- 172.21.15.26:0/1449022350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4720009e80 tx=0x7f472000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff88bd3d56610487 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f4727fff700 1 -- 172.21.15.26:0/1449022350 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472000e070 con 0x7f4728106fc0 2024-03-24T10:34:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f4727fff700 1 -- 172.21.15.26:0/1449022350 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f472000b8c0 con 0x7f4728106fc0 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.339+0000 7f4727fff700 1 -- 172.21.15.26:0/1449022350 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4720004440 con 0x7f4728106fc0 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 -- 172.21.15.26:0/1449022350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 msgr2=0x7f47281093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 --2- 172.21.15.26:0/1449022350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4720009e80 tx=0x7f472000b330 comp rx=0 tx=0).stop 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 -- 172.21.15.26:0/1449022350 shutdown_connections 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 --2- 172.21.15.26:0/1449022350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281093b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 -- 172.21.15.26:0/1449022350 >> 172.21.15.26:0/1449022350 conn(0x7f4728100a10 msgr2=0x7f4728102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 -- 172.21.15.26:0/1449022350 shutdown_connections 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.340+0000 7f472f95f700 1 -- 172.21.15.26:0/1449022350 wait complete. 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 Processor -- start 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 -- start start 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728106d30 con 0x7f4728106fc0 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472d6fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.119 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472d6fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35762/0 (socket says 172.21.15.26:35762) 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472d6fb700 1 -- 172.21.15.26:0/752299932 learned_addr learned my addr 172.21.15.26:0/752299932 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472d6fb700 1 -- 172.21.15.26:0/752299932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720009b30 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472d6fb700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4720003c50 tx=0x7f4720003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472000e040 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4728104f70 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f472000bc40 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47200125e0 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.341+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4728105470 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.342+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f4720019040 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.342+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472810a670 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.342+0000 7f47267fc700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 0x7f471003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.343+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f472004c120 con 0x7f4728106fc0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.343+0000 7f472cefa700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 0x7f471003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.120 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.343+0000 7f472cefa700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 0x7f471003abe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f471c006fd0 tx=0x7f471c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.347+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f472001a570 con 0x7f4728106fc0 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.564+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f4728101c10 con 0x7f4710038720 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.569+0000 7f47267fc700 1 -- 172.21.15.26:0/752299932 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4728101c10 con 0x7f4710038720 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 msgr2=0x7f471003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 0x7f471003abe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f471c006fd0 tx=0x7f471c006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 msgr2=0x7f47281067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4720003c50 tx=0x7f4720003d30 comp rx=0 tx=0).stop 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 shutdown_connections 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4710038720 0x7f471003abe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 --2- 172.21.15.26:0/752299932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4728106fc0 0x7f47281067f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 >> 172.21.15.26:0/752299932 conn(0x7f4728100a10 msgr2=0x7f4728101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 shutdown_connections 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:41.571+0000 7f472f95f700 1 -- 172.21.15.26:0/752299932 wait complete. 2024-03-24T10:34:43.121 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying mgr service with default placement... 2024-03-24T10:34:44.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:43 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:44.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:43 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:44.098 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:43 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7effa064e700 1 Processor -- start 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7effa064e700 1 -- start start 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7effa064e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7effa064e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9807a820 con 0x7eff9807bd90 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7eff9e3ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7eff9e3ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57612/0 (socket says 172.21.15.26:57612) 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7eff9e3ea700 1 -- 172.21.15.26:0/210413828 learned_addr learned my addr 172.21.15.26:0/210413828 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.563+0000 7eff9e3ea700 1 -- 172.21.15.26:0/210413828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9807a960 con 0x7eff9807bd90 2024-03-24T10:34:44.399 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.564+0000 7eff9e3ea700 1 --2- 172.21.15.26:0/210413828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7eff940098a0 tx=0x7eff94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112b00c377d29646 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.400 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.564+0000 7eff9d3e8700 1 -- 172.21.15.26:0/210413828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff94004180 con 0x7eff9807bd90 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.564+0000 7eff9d3e8700 1 -- 172.21.15.26:0/210413828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff940042e0 con 0x7eff9807bd90 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.564+0000 7eff9d3e8700 1 -- 172.21.15.26:0/210413828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff94011b10 con 0x7eff9807bd90 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- 172.21.15.26:0/210413828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 msgr2=0x7eff9807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 --2- 172.21.15.26:0/210413828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7eff940098a0 tx=0x7eff94009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- 172.21.15.26:0/210413828 shutdown_connections 2024-03-24T10:34:44.401 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 --2- 172.21.15.26:0/210413828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9807a2e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- 172.21.15.26:0/210413828 >> 172.21.15.26:0/210413828 conn(0x7eff98101b50 msgr2=0x7eff98103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- 172.21.15.26:0/210413828 shutdown_connections 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- 172.21.15.26:0/210413828 wait complete. 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 Processor -- start 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- start start 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.565+0000 7effa064e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9819c960 con 0x7eff9807bd90 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff9e3ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff9e3ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57624/0 (socket says 172.21.15.26:57624) 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff9e3ea700 1 -- 172.21.15.26:0/3283946954 learned_addr learned my addr 172.21.15.26:0/3283946954 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff9e3ea700 1 -- 172.21.15.26:0/3283946954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff94009550 con 0x7eff9807bd90 2024-03-24T10:34:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff9e3ea700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7eff940117b0 tx=0x7eff94011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff940038b0 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9819cb60 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff94003a10 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff940246e0 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.566+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9819d000 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.567+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7eff9402d3f0 con 0x7eff9807bd90 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.567+0000 7eff8f7fe700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 0x7eff8403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.403 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.567+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff9404c7d0 con 0x7eff9807bd90 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.567+0000 7eff9dbe9700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 0x7eff8403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.567+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff980623e0 con 0x7eff9807bd90 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.568+0000 7eff9dbe9700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 0x7eff8403ac30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff88006fd0 tx=0x7eff88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.572+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff9402d6f0 con 0x7eff9807bd90 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.792+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7eff98102e10 con 0x7eff84038770 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.797+0000 7eff8f7fe700 1 -- 172.21.15.26:0/3283946954 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7eff98102e10 con 0x7eff84038770 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 msgr2=0x7eff8403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 0x7eff8403ac30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff88006fd0 tx=0x7eff88006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 msgr2=0x7eff9819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7eff940117b0 tx=0x7eff94011890 comp rx=0 tx=0).stop 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 shutdown_connections 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7eff84038770 0x7eff8403ac30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 --2- 172.21.15.26:0/3283946954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff9807bd90 0x7eff9819c420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.800+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 >> 172.21.15.26:0/3283946954 conn(0x7eff98101b50 msgr2=0x7eff98102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:44.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.801+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 shutdown_connections 2024-03-24T10:34:44.405 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:43.801+0000 7effa064e700 1 -- 172.21.15.26:0/3283946954 wait complete. 2024-03-24T10:34:44.405 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying crash service with default placement... 2024-03-24T10:34:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:44 smithi026 ceph-mon[29210]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:44 smithi026 ceph-mon[29210]: Saving service mgr spec with placement count:2 2024-03-24T10:34:46.074 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-24T10:34:46.074 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f4dd73700 1 Processor -- start 2024-03-24T10:34:46.074 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f4dd73700 1 -- start start 2024-03-24T10:34:46.074 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f4dd73700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.074 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f4dd73700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4807a8b0 con 0x7f4f4807be30 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57630/0 (socket says 172.21.15.26:57630) 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.815+0000 7f4f477fe700 1 -- 172.21.15.26:0/1978420180 learned_addr learned my addr 172.21.15.26:0/1978420180 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.816+0000 7f4f477fe700 1 -- 172.21.15.26:0/1978420180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4807a9f0 con 0x7f4f4807be30 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.816+0000 7f4f477fe700 1 --2- 172.21.15.26:0/1978420180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009e80 tx=0x7f4f4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14eecababec4a22e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.816+0000 7f4f467fc700 1 -- 172.21.15.26:0/1978420180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4000e070 con 0x7f4f4807be30 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.816+0000 7f4f467fc700 1 -- 172.21.15.26:0/1978420180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f4000b8c0 con 0x7f4f4807be30 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.816+0000 7f4f467fc700 1 -- 172.21.15.26:0/1978420180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f400044d0 con 0x7f4f4807be30 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 -- 172.21.15.26:0/1978420180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 msgr2=0x7f4f4807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/1978420180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009e80 tx=0x7f4f4000b330 comp rx=0 tx=0).stop 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 -- 172.21.15.26:0/1978420180 shutdown_connections 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/1978420180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4807a2e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 -- 172.21.15.26:0/1978420180 >> 172.21.15.26:0/1978420180 conn(0x7f4f48101b10 msgr2=0x7f4f48103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:46.075 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 -- 172.21.15.26:0/1978420180 shutdown_connections 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 -- 172.21.15.26:0/1978420180 wait complete. 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.817+0000 7f4f4dd73700 1 Processor -- start 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f4dd73700 1 -- start start 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f4dd73700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f4dd73700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4819c8d0 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57640/0 (socket says 172.21.15.26:57640) 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f477fe700 1 -- 172.21.15.26:0/2168348394 learned_addr learned my addr 172.21.15.26:0/2168348394 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f477fe700 1 -- 172.21.15.26:0/2168348394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f40009b30 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f477fe700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f400072a0 tx=0x7f4f4000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4000e040 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f40003dc0 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4819cad0 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.818+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f40012750 con 0x7f4f4807be30 2024-03-24T10:34:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.819+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4819cfd0 con 0x7f4f4807be30 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.819+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f4f40019040 con 0x7f4f4807be30 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.819+0000 7f4f44ff9700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 0x7f4f3403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.820+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4f4004c2b0 con 0x7f4f4807be30 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.820+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4810c870 con 0x7f4f4807be30 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.820+0000 7f4f46ffd700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 0x7f4f3403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.820+0000 7f4f46ffd700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 0x7f4f3403abe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f30006fd0 tx=0x7f4f30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:44.825+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f4001f080 con 0x7f4f4807be30 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.040+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f4819fc20 con 0x7f4f34038720 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.044+0000 7f4f44ff9700 1 -- 172.21.15.26:0/2168348394 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4f4819fc20 con 0x7f4f34038720 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 msgr2=0x7f4f3403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 0x7f4f3403abe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f30006fd0 tx=0x7f4f30006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 msgr2=0x7f4f4819c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f400072a0 tx=0x7f4f4000be00 comp rx=0 tx=0).stop 2024-03-24T10:34:46.077 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 shutdown_connections 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f4f34038720 0x7f4f3403abe0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.046+0000 7f4f4dd73700 1 --2- 172.21.15.26:0/2168348394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f4807be30 0x7f4f4819c390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.047+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 >> 172.21.15.26:0/2168348394 conn(0x7f4f48101b10 msgr2=0x7f4f481026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.047+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 shutdown_connections 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:45.047+0000 7f4f4dd73700 1 -- 172.21.15.26:0/2168348394 wait complete. 2024-03-24T10:34:46.078 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying ceph-exporter service with default placement... 2024-03-24T10:34:46.360 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:46 smithi026 ceph-mon[29210]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:46.360 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:46 smithi026 ceph-mon[29210]: Saving service crash spec with placement * 2024-03-24T10:34:46.360 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:46 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:46.360 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:46 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.580+0000 7f04c99f3700 1 Processor -- start 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.580+0000 7f04c99f3700 1 -- start start 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c99f3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c99f3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4106ee0 con 0x7f04c41064f0 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57654/0 (socket says 172.21.15.26:57654) 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c2ffd700 1 -- 172.21.15.26:0/3897494149 learned_addr learned my addr 172.21.15.26:0/3897494149 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:47.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c2ffd700 1 -- 172.21.15.26:0/3897494149 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c41076a0 con 0x7f04c41064f0 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c2ffd700 1 --2- 172.21.15.26:0/3897494149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04b40098a0 tx=0x7f04b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c117ea590dd89f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c1ffb700 1 -- 172.21.15.26:0/3897494149 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4004180 con 0x7f04c41064f0 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c1ffb700 1 -- 172.21.15.26:0/3897494149 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04b40042e0 con 0x7f04c41064f0 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.581+0000 7f04c1ffb700 1 -- 172.21.15.26:0/3897494149 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4011b10 con 0x7f04c41064f0 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 -- 172.21.15.26:0/3897494149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 msgr2=0x7f04c4106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 --2- 172.21.15.26:0/3897494149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04b40098a0 tx=0x7f04b4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 -- 172.21.15.26:0/3897494149 shutdown_connections 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 --2- 172.21.15.26:0/3897494149 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c4106910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 -- 172.21.15.26:0/3897494149 >> 172.21.15.26:0/3897494149 conn(0x7f04c4101b70 msgr2=0x7f04c4103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 -- 172.21.15.26:0/3897494149 shutdown_connections 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.582+0000 7f04c99f3700 1 -- 172.21.15.26:0/3897494149 wait complete. 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.583+0000 7f04c99f3700 1 Processor -- start 2024-03-24T10:34:47.365 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.583+0000 7f04c99f3700 1 -- start start 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.583+0000 7f04c99f3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.583+0000 7f04c99f3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c407c040 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57660/0 (socket says 172.21.15.26:57660) 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c2ffd700 1 -- 172.21.15.26:0/399390201 learned_addr learned my addr 172.21.15.26:0/399390201 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c2ffd700 1 -- 172.21.15.26:0/399390201 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009550 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c2ffd700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009fb0 tx=0x7f04b4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b40038b0 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c407a280 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04b4003a10 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4019770 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.584+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c407a720 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.585+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f04b40198d0 con 0x7f04c41064f0 2024-03-24T10:34:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.585+0000 7f04c89f1700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 0x7f04ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.585+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f04b404d860 con 0x7f04c41064f0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.585+0000 7f04c27fc700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 0x7f04ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.586+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c40623e0 con 0x7f04c41064f0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.586+0000 7f04c27fc700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 0x7f04ac03aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04b8006fd0 tx=0x7f04b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.591+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f04b4022d70 con 0x7f04c41064f0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.816+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c407b030 con 0x7f04ac0386e0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.821+0000 7f04c89f1700 1 -- 172.21.15.26:0/399390201 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f04c407b030 con 0x7f04ac0386e0 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 msgr2=0x7f04ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 0x7f04ac03aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04b8006fd0 tx=0x7f04b8006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 msgr2=0x7f04c407bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009fb0 tx=0x7f04b4011890 comp rx=0 tx=0).stop 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 shutdown_connections 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f04ac0386e0 0x7f04ac03aba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 --2- 172.21.15.26:0/399390201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c41064f0 0x7f04c407bb00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.368 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 >> 172.21.15.26:0/399390201 conn(0x7f04c4101b70 msgr2=0x7f04c41026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:47.368 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 shutdown_connections 2024-03-24T10:34:47.368 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:46.823+0000 7f04c99f3700 1 -- 172.21.15.26:0/399390201 wait complete. 2024-03-24T10:34:47.368 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying prometheus service with default placement... 2024-03-24T10:34:47.934 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:47 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:47.934 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:47 smithi026 ceph-mon[29210]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:47.934 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:47 smithi026 ceph-mon[29210]: Saving service ceph-exporter spec with placement * 2024-03-24T10:34:47.935 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:47 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:49.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f3b241700 1 Processor -- start 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f3b241700 1 -- start start 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f3b241700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f3b241700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34074590 con 0x7f8f34106f90 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f38fdd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f38fdd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57668/0 (socket says 172.21.15.26:57668) 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.834+0000 7f8f38fdd700 1 -- 172.21.15.26:0/3514861902 learned_addr learned my addr 172.21.15.26:0/3514861902 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.835+0000 7f8f38fdd700 1 -- 172.21.15.26:0/3514861902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f341098c0 con 0x7f8f34106f90 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.835+0000 7f8f38fdd700 1 --2- 172.21.15.26:0/3514861902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f240098a0 tx=0x7f8f24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3ab70ebea76009 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.835+0000 7f8f337fe700 1 -- 172.21.15.26:0/3514861902 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f24004180 con 0x7f8f34106f90 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.835+0000 7f8f337fe700 1 -- 172.21.15.26:0/3514861902 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f240042e0 con 0x7f8f34106f90 2024-03-24T10:34:49.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.835+0000 7f8f337fe700 1 -- 172.21.15.26:0/3514861902 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f24011b10 con 0x7f8f34106f90 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 -- 172.21.15.26:0/3514861902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 msgr2=0x7f8f34109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 --2- 172.21.15.26:0/3514861902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f240098a0 tx=0x7f8f24009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 -- 172.21.15.26:0/3514861902 shutdown_connections 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 --2- 172.21.15.26:0/3514861902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f34109380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 -- 172.21.15.26:0/3514861902 >> 172.21.15.26:0/3514861902 conn(0x7f8f341009c0 msgr2=0x7f8f34102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 -- 172.21.15.26:0/3514861902 shutdown_connections 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.836+0000 7f8f3b241700 1 -- 172.21.15.26:0/3514861902 wait complete. 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f3b241700 1 Processor -- start 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f3b241700 1 -- start start 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f3b241700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f3b241700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3419a750 con 0x7f8f34106f90 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f38fdd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f38fdd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57676/0 (socket says 172.21.15.26:57676) 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f38fdd700 1 -- 172.21.15.26:0/1331651956 learned_addr learned my addr 172.21.15.26:0/1331651956 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f38fdd700 1 -- 172.21.15.26:0/1331651956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f24009550 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.837+0000 7f8f38fdd700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f240117b0 tx=0x7f8f24011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f240038b0 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f24003a10 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3419a950 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f24024780 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3419adf0 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.838+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f8f240248e0 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.839+0000 7f8f31ffb700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 0x7f8f2003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.839+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f2404c750 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.839+0000 7f8f33fff700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 0x7f8f2003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.839+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f340623e0 con 0x7f8f34106f90 2024-03-24T10:34:49.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.840+0000 7f8f33fff700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 0x7f8f2003ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c006fd0 tx=0x7f8f1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:47.847+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f24024b90 con 0x7f8f34106f90 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.066+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f34101c30 con 0x7f8f20038770 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.070+0000 7f8f31ffb700 1 -- 172.21.15.26:0/1331651956 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8f34101c30 con 0x7f8f20038770 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 msgr2=0x7f8f2003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 0x7f8f2003ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c006fd0 tx=0x7f8f1c006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 msgr2=0x7f8f3419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f240117b0 tx=0x7f8f24011890 comp rx=0 tx=0).stop 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 shutdown_connections 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f8f20038770 0x7f8f2003ac30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 --2- 172.21.15.26:0/1331651956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8f34106f90 0x7f8f3419a210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.072+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 >> 172.21.15.26:0/1331651956 conn(0x7f8f341009c0 msgr2=0x7f8f34101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.073+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 shutdown_connections 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:48.073+0000 7f8f3b241700 1 -- 172.21.15.26:0/1331651956 wait complete. 2024-03-24T10:34:49.037 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying grafana service with default placement... 2024-03-24T10:34:49.285 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:49 smithi026 ceph-mon[29210]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:49.285 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:49 smithi026 ceph-mon[29210]: Saving service prometheus spec with placement count:1 2024-03-24T10:34:49.285 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:49 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa323eb4700 1 Processor -- start 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa323eb4700 1 -- start start 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa323eb4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa323eb4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c074590 con 0x7fa31c106f90 2024-03-24T10:34:50.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa321c50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa321c50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57686/0 (socket says 172.21.15.26:57686) 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.495+0000 7fa321c50700 1 -- 172.21.15.26:0/1032866147 learned_addr learned my addr 172.21.15.26:0/1032866147 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa321c50700 1 -- 172.21.15.26:0/1032866147 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c1098c0 con 0x7fa31c106f90 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa321c50700 1 --2- 172.21.15.26:0/1032866147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3180098a0 tx=0x7fa318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563b012a1ea83330 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa320c4e700 1 -- 172.21.15.26:0/1032866147 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa318004180 con 0x7fa31c106f90 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa320c4e700 1 -- 172.21.15.26:0/1032866147 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3180042e0 con 0x7fa31c106f90 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa320c4e700 1 -- 172.21.15.26:0/1032866147 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa318011b10 con 0x7fa31c106f90 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa323eb4700 1 -- 172.21.15.26:0/1032866147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 msgr2=0x7fa31c109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.496+0000 7fa323eb4700 1 --2- 172.21.15.26:0/1032866147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3180098a0 tx=0x7fa318009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- 172.21.15.26:0/1032866147 shutdown_connections 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 --2- 172.21.15.26:0/1032866147 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c109380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- 172.21.15.26:0/1032866147 >> 172.21.15.26:0/1032866147 conn(0x7fa31c1009c0 msgr2=0x7fa31c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- 172.21.15.26:0/1032866147 shutdown_connections 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- 172.21.15.26:0/1032866147 wait complete. 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 Processor -- start 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- start start 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.497+0000 7fa323eb4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c19a750 con 0x7fa31c106f90 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa321c50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa321c50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57690/0 (socket says 172.21.15.26:57690) 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa321c50700 1 -- 172.21.15.26:0/546847307 learned_addr learned my addr 172.21.15.26:0/546847307 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa321c50700 1 -- 172.21.15.26:0/546847307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318009550 con 0x7fa31c106f90 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa321c50700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3180117b0 tx=0x7fa318011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3180038b0 con 0x7fa31c106f90 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa318003a10 con 0x7fa31c106f90 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c19a950 con 0x7fa31c106f90 2024-03-24T10:34:50.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa318024780 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.498+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c19adf0 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.499+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fa3180248e0 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.499+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c10a650 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.499+0000 7fa312ffd700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 0x7fa30803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.499+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa31804c910 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.499+0000 7fa32144f700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 0x7fa30803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.500+0000 7fa32144f700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 0x7fa30803ac30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa30c006fd0 tx=0x7fa30c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.504+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa318024b90 con 0x7fa31c106f90 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.726+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa31c101c30 con 0x7fa308038770 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.731+0000 7fa312ffd700 1 -- 172.21.15.26:0/546847307 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa31c101c30 con 0x7fa308038770 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 msgr2=0x7fa30803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 0x7fa30803ac30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa30c006fd0 tx=0x7fa30c006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 msgr2=0x7fa31c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3180117b0 tx=0x7fa318011890 comp rx=0 tx=0).stop 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 shutdown_connections 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa308038770 0x7fa30803ac30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 --2- 172.21.15.26:0/546847307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa31c106f90 0x7fa31c19a210 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 >> 172.21.15.26:0/546847307 conn(0x7fa31c1009c0 msgr2=0x7fa31c101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 shutdown_connections 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:49.733+0000 7fa323eb4700 1 -- 172.21.15.26:0/546847307 wait complete. 2024-03-24T10:34:50.414 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying node-exporter service with default placement... 2024-03-24T10:34:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:50 smithi026 ceph-mon[29210]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:50 smithi026 ceph-mon[29210]: Saving service grafana spec with placement count:1 2024-03-24T10:34:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:50 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:50 smithi026 ceph-mon[29210]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:34:51.613 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-24T10:34:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.803+0000 7feb6288c700 1 Processor -- start 2024-03-24T10:34:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.803+0000 7feb6288c700 1 -- start start 2024-03-24T10:34:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb6288c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb6288c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c07a820 con 0x7feb5c07bde0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57694/0 (socket says 172.21.15.26:57694) 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5bfff700 1 -- 172.21.15.26:0/4039385309 learned_addr learned my addr 172.21.15.26:0/4039385309 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5bfff700 1 -- 172.21.15.26:0/4039385309 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c07a960 con 0x7feb5c07bde0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5bfff700 1 --2- 172.21.15.26:0/4039385309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0098a0 tx=0x7feb4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4998fb0c8eee933 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5affd700 1 -- 172.21.15.26:0/4039385309 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4c004180 con 0x7feb5c07bde0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5affd700 1 -- 172.21.15.26:0/4039385309 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb4c0042e0 con 0x7feb5c07bde0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.804+0000 7feb5affd700 1 -- 172.21.15.26:0/4039385309 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4c011b10 con 0x7feb5c07bde0 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 -- 172.21.15.26:0/4039385309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 msgr2=0x7feb5c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 --2- 172.21.15.26:0/4039385309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0098a0 tx=0x7feb4c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 -- 172.21.15.26:0/4039385309 shutdown_connections 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 --2- 172.21.15.26:0/4039385309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c07a2e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.615 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 -- 172.21.15.26:0/4039385309 >> 172.21.15.26:0/4039385309 conn(0x7feb5c101b10 msgr2=0x7feb5c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 -- 172.21.15.26:0/4039385309 shutdown_connections 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.805+0000 7feb6288c700 1 -- 172.21.15.26:0/4039385309 wait complete. 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb6288c700 1 Processor -- start 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb6288c700 1 -- start start 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb6288c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb6288c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c19c9a0 con 0x7feb5c07bde0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57698/0 (socket says 172.21.15.26:57698) 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb5bfff700 1 -- 172.21.15.26:0/1690378220 learned_addr learned my addr 172.21.15.26:0/1690378220 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.806+0000 7feb5bfff700 1 -- 172.21.15.26:0/1690378220 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c009550 con 0x7feb5c07bde0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb5bfff700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feb4c004210 tx=0x7feb4c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4c0038b0 con 0x7feb5c07bde0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c19cba0 con 0x7feb5c07bde0 2024-03-24T10:34:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb4c003a10 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4c024650 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.807+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c19d040 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.808+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7feb4c024910 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.808+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c10c830 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.808+0000 7feb597fa700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 0x7feb4403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.808+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb4c04c5e0 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.808+0000 7feb5b7fe700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 0x7feb4403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.809+0000 7feb5b7fe700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 0x7feb4403abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb50006fd0 tx=0x7feb50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:50.813+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb4c024bc0 con 0x7feb5c07bde0 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.027+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7feb5c19fd50 con 0x7feb44038720 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.032+0000 7feb597fa700 1 -- 172.21.15.26:0/1690378220 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7feb5c19fd50 con 0x7feb44038720 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 msgr2=0x7feb4403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 0x7feb4403abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb50006fd0 tx=0x7feb50006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 msgr2=0x7feb5c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feb4c004210 tx=0x7feb4c011940 comp rx=0 tx=0).stop 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 shutdown_connections 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7feb44038720 0x7feb4403abe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 --2- 172.21.15.26:0/1690378220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb5c07bde0 0x7feb5c19c460 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 >> 172.21.15.26:0/1690378220 conn(0x7feb5c101b10 msgr2=0x7feb5c1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 shutdown_connections 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:51.034+0000 7feb6288c700 1 -- 172.21.15.26:0/1690378220 wait complete. 2024-03-24T10:34:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying alertmanager service with default placement... 2024-03-24T10:34:52.184 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:52 smithi026 ceph-mon[29210]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:52.184 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:52 smithi026 ceph-mon[29210]: Saving service node-exporter spec with placement * 2024-03-24T10:34:52.184 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:52 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd9d4d700 1 Processor -- start 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd9d4d700 1 -- start start 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd9d4d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd9d4d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4079e90 con 0x7f2dd4106f60 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57712/0 (socket says 172.21.15.26:57712) 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd37fe700 1 -- 172.21.15.26:0/4162503245 learned_addr learned my addr 172.21.15.26:0/4162503245 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd37fe700 1 -- 172.21.15.26:0/4162503245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4109890 con 0x7f2dd4106f60 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.209+0000 7f2dd37fe700 1 --2- 172.21.15.26:0/4162503245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40098a0 tx=0x7f2dc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862f2b196cb2d70e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd27fc700 1 -- 172.21.15.26:0/4162503245 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc4004180 con 0x7f2dd4106f60 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd27fc700 1 -- 172.21.15.26:0/4162503245 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dc40042e0 con 0x7f2dd4106f60 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd27fc700 1 -- 172.21.15.26:0/4162503245 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc4011b10 con 0x7f2dd4106f60 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/4162503245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 msgr2=0x7f2dd4109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/4162503245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40098a0 tx=0x7f2dc4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/4162503245 shutdown_connections 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/4162503245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd4109350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/4162503245 >> 172.21.15.26:0/4162503245 conn(0x7f2dd41009b0 msgr2=0x7f2dd4102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/4162503245 shutdown_connections 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.210+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/4162503245 wait complete. 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd9d4d700 1 Processor -- start 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd9d4d700 1 -- start start 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd9d4d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd9d4d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd419a740 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57720/0 (socket says 172.21.15.26:57720) 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.211+0000 7f2dd37fe700 1 -- 172.21.15.26:0/1497655471 learned_addr learned my addr 172.21.15.26:0/1497655471 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd37fe700 1 -- 172.21.15.26:0/1497655471 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4009550 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd37fe700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009fb0 tx=0x7f2dc4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc40038b0 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dc4003a10 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd419a940 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc4019750 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.212+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd419ade0 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.213+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f2dc40199c0 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.213+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd410a680 con 0x7f2dd4106f60 2024-03-24T10:34:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.213+0000 7f2dd0ff9700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 0x7f2dbc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.213+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2dc404c9a0 con 0x7f2dd4106f60 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.213+0000 7f2dd2ffd700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 0x7f2dbc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.214+0000 7f2dd2ffd700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 0x7f2dbc03aba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8006fd0 tx=0x7f2dc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.218+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2dc4010380 con 0x7f2dd4106f60 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.432+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dd4101ba0 con 0x7f2dbc0386e0 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.437+0000 7f2dd0ff9700 1 -- 172.21.15.26:0/1497655471 <== mgr.14120 v2:172.21.15.26:6800/3234553684 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f2dd4101ba0 con 0x7f2dbc0386e0 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 msgr2=0x7f2dbc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.999 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 0x7f2dbc03aba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8006fd0 tx=0x7f2dc8006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 msgr2=0x7f2dd419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009fb0 tx=0x7f2dc4011890 comp rx=0 tx=0).stop 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 shutdown_connections 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f2dbc0386e0 0x7f2dbc03aba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 --2- 172.21.15.26:0/1497655471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd4106f60 0x7f2dd419a200 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.439+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 >> 172.21.15.26:0/1497655471 conn(0x7f2dd41009b0 msgr2=0x7f2dd4101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.440+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 shutdown_connections 2024-03-24T10:34:53.000 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:52.440+0000 7f2dd9d4d700 1 -- 172.21.15.26:0/1497655471 wait complete. 2024-03-24T10:34:53.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:53 smithi026 ceph-mon[29210]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:34:53.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:53 smithi026 ceph-mon[29210]: from='mgr.14120 172.21.15.26:0/1305276224' entity='mgr.smithi026.bbjcru' 2024-03-24T10:34:54.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:54 smithi026 ceph-mon[29210]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:54.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:54 smithi026 ceph-mon[29210]: Saving service alertmanager spec with placement count:1 2024-03-24T10:34:54.194 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:54 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2152684208' entity='client.admin' 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.417+0000 7fca8e55a700 1 Processor -- start 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.417+0000 7fca8e55a700 1 -- start start 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca8e55a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca8e55a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca881073e0 con 0x7fca881069f0 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47584/0 (socket says 172.21.15.26:47584) 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca87fff700 1 -- 172.21.15.26:0/237934553 learned_addr learned my addr 172.21.15.26:0/237934553 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca87fff700 1 -- 172.21.15.26:0/237934553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca88105650 con 0x7fca881069f0 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca87fff700 1 --2- 172.21.15.26:0/237934553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fca7800bb40 tx=0x7fca7800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f67bcf129aea15d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.418+0000 7fca86ffd700 1 -- 172.21.15.26:0/237934553 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca7800dc00 con 0x7fca881069f0 2024-03-24T10:34:54.201 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca86ffd700 1 -- 172.21.15.26:0/237934553 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca780044f0 con 0x7fca881069f0 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca86ffd700 1 -- 172.21.15.26:0/237934553 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca78010470 con 0x7fca881069f0 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 -- 172.21.15.26:0/237934553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 msgr2=0x7fca88106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 --2- 172.21.15.26:0/237934553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fca7800bb40 tx=0x7fca7800be50 comp rx=0 tx=0).stop 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 -- 172.21.15.26:0/237934553 shutdown_connections 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 --2- 172.21.15.26:0/237934553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca88106e10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 -- 172.21.15.26:0/237934553 >> 172.21.15.26:0/237934553 conn(0x7fca881001a0 msgr2=0x7fca881025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 -- 172.21.15.26:0/237934553 shutdown_connections 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.419+0000 7fca8e55a700 1 -- 172.21.15.26:0/237934553 wait complete. 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca8e55a700 1 Processor -- start 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca8e55a700 1 -- start start 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca8e55a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca8e55a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8819c9b0 con 0x7fca881069f0 2024-03-24T10:34:54.202 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47600/0 (socket says 172.21.15.26:47600) 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca87fff700 1 -- 172.21.15.26:0/2152684208 learned_addr learned my addr 172.21.15.26:0/2152684208 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.420+0000 7fca87fff700 1 -- 172.21.15.26:0/2152684208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7800b7f0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca87fff700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca78003cb0 tx=0x7fca78004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca780106a0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8819cbb0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca78010cc0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca78021ac0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.421+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8819cff0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fca78021d40 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca88104540 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca857fa700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 0x7fca7003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fca7804c9a0 con 0x7fca881069f0 2024-03-24T10:34:54.203 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca877fe700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 0x7fca7003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.422+0000 7fca877fe700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 0x7fca7003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca7c006fd0 tx=0x7fca7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.427+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fca7800e770 con 0x7fca881069f0 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.625+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fca881917a0 con 0x7fca881069f0 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.628+0000 7fca857fa700 1 -- 172.21.15.26:0/2152684208 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fca78029360 con 0x7fca881069f0 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 msgr2=0x7fca7003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 0x7fca7003abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca7c006fd0 tx=0x7fca7c006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 msgr2=0x7fca8819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca78003cb0 tx=0x7fca78004070 comp rx=0 tx=0).stop 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 shutdown_connections 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fca70038720 0x7fca7003abe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 --2- 172.21.15.26:0/2152684208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca881069f0 0x7fca8819c470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 >> 172.21.15.26:0/2152684208 conn(0x7fca881001a0 msgr2=0x7fca8818e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.204 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 shutdown_connections 2024-03-24T10:34:54.205 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:53.631+0000 7fca8e55a700 1 -- 172.21.15.26:0/2152684208 wait complete. 2024-03-24T10:34:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:55 smithi026 ceph-mon[29210]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:34:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:55 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/186504526' entity='client.admin' 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b8980b700 1 Processor -- start 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b8980b700 1 -- start start 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b8980b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b8980b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b84079e30 con 0x7f7b84106fc0 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47610/0 (socket says 172.21.15.26:47610) 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b82ffd700 1 -- 172.21.15.26:0/3473830402 learned_addr learned my addr 172.21.15.26:0/3473830402 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.653+0000 7f7b82ffd700 1 -- 172.21.15.26:0/3473830402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b841098f0 con 0x7f7b84106fc0 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b82ffd700 1 --2- 172.21.15.26:0/3473830402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b740098a0 tx=0x7f7b74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b48b24ed642b7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b81ffb700 1 -- 172.21.15.26:0/3473830402 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b74004180 con 0x7f7b84106fc0 2024-03-24T10:34:55.550 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b81ffb700 1 -- 172.21.15.26:0/3473830402 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b740042e0 con 0x7f7b84106fc0 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b81ffb700 1 -- 172.21.15.26:0/3473830402 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b74011b10 con 0x7f7b84106fc0 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 -- 172.21.15.26:0/3473830402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 msgr2=0x7f7b841093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 --2- 172.21.15.26:0/3473830402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b740098a0 tx=0x7f7b74009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 -- 172.21.15.26:0/3473830402 shutdown_connections 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 --2- 172.21.15.26:0/3473830402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b841093b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 -- 172.21.15.26:0/3473830402 >> 172.21.15.26:0/3473830402 conn(0x7f7b841009d0 msgr2=0x7f7b84102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 -- 172.21.15.26:0/3473830402 shutdown_connections 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.654+0000 7f7b8980b700 1 -- 172.21.15.26:0/3473830402 wait complete. 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b8980b700 1 Processor -- start 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b8980b700 1 -- start start 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b8980b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b8980b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8419a740 con 0x7f7b84106fc0 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.551 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47626/0 (socket says 172.21.15.26:47626) 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.655+0000 7f7b82ffd700 1 -- 172.21.15.26:0/186504526 learned_addr learned my addr 172.21.15.26:0/186504526 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b82ffd700 1 -- 172.21.15.26:0/186504526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009550 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b82ffd700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fb0 tx=0x7f7b74011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b740038b0 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b74003a10 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8419a940 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b74019750 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.656+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8419ade0 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f7b740198b0 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b88809700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 0x7f7b6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7b7404c7d0 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8410a680 con 0x7f7b84106fc0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b827fc700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 0x7f7b6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.552 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.657+0000 7f7b827fc700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 0x7f7b6c03abf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.662+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b74019bb0 con 0x7f7b84106fc0 2024-03-24T10:34:55.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.862+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7b8419da90 con 0x7f7b84106fc0 2024-03-24T10:34:55.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.865+0000 7f7b88809700 1 -- 172.21.15.26:0/186504526 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7b8419da90 con 0x7f7b84106fc0 2024-03-24T10:34:55.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.867+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 msgr2=0x7f7b6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.867+0000 7f7b8980b700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 0x7f7b6c03abf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:55.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.867+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 msgr2=0x7f7b8419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.867+0000 7f7b8980b700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fb0 tx=0x7f7b74011890 comp rx=0 tx=0).stop 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 shutdown_connections 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f7b6c038730 0x7f7b6c03abf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 --2- 172.21.15.26:0/186504526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b84106fc0 0x7f7b8419a200 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 >> 172.21.15.26:0/186504526 conn(0x7f7b841009d0 msgr2=0x7f7b84101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 shutdown_connections 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:54.868+0000 7f7b8980b700 1 -- 172.21.15.26:0/186504526 wait complete. 2024-03-24T10:34:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling the dashboard module... 2024-03-24T10:34:56.680 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:56 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1480465146' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T10:34:57.032 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.981+0000 7f0c5a61a700 1 Processor -- start 2024-03-24T10:34:57.032 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.981+0000 7f0c5a61a700 1 -- start start 2024-03-24T10:34:57.032 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c5a61a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.032 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c5a61a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54106ef0 con 0x7f0c54106500 2024-03-24T10:34:57.032 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47640/0 (socket says 172.21.15.26:47640) 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c53fff700 1 -- 172.21.15.26:0/94313079 learned_addr learned my addr 172.21.15.26:0/94313079 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c53fff700 1 -- 172.21.15.26:0/94313079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c541076b0 con 0x7f0c54106500 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c53fff700 1 --2- 172.21.15.26:0/94313079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098a0 tx=0x7f0c44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448a40c60f1d5823 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.982+0000 7f0c537fe700 1 -- 172.21.15.26:0/94313079 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c44004180 con 0x7f0c54106500 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c537fe700 1 -- 172.21.15.26:0/94313079 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c440042e0 con 0x7f0c54106500 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c537fe700 1 -- 172.21.15.26:0/94313079 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c44011b10 con 0x7f0c54106500 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 -- 172.21.15.26:0/94313079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 msgr2=0x7f0c54106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/94313079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098a0 tx=0x7f0c44009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.033 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 -- 172.21.15.26:0/94313079 shutdown_connections 2024-03-24T10:34:57.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/94313079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c54106920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 -- 172.21.15.26:0/94313079 >> 172.21.15.26:0/94313079 conn(0x7f0c54101b40 msgr2=0x7f0c54103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 -- 172.21.15.26:0/94313079 shutdown_connections 2024-03-24T10:34:57.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.983+0000 7f0c5a61a700 1 -- 172.21.15.26:0/94313079 wait complete. 2024-03-24T10:34:57.034 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c5a61a700 1 Processor -- start 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c5a61a700 1 -- start start 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c5a61a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c5a61a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5407c040 con 0x7f0c54106500 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47646/0 (socket says 172.21.15.26:47646) 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c53fff700 1 -- 172.21.15.26:0/1480465146 learned_addr learned my addr 172.21.15.26:0/1480465146 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.984+0000 7f0c53fff700 1 -- 172.21.15.26:0/1480465146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44009550 con 0x7f0c54106500 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c53fff700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009fb0 tx=0x7f0c44011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.035 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c440038b0 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c44003a10 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5407a280 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c44019770 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.985+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5407a720 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f0c440198d0 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c540623e0 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c51ffb700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 0x7f0c4003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0c4404c810 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c4bfff700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 0x7f0c4003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.986+0000 7f0c4bfff700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 0x7f0c4003aba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c006fd0 tx=0x7f0c3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:55.991+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0c44022d70 con 0x7f0c54106500 2024-03-24T10:34:57.036 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.253+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0c5410a8e0 con 0x7f0c54106500 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.306+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f0c4401ac10 con 0x7f0c54106500 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.307+0000 7f0c51ffb700 1 -- 172.21.15.26:0/1480465146 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0c4402bbd0 con 0x7f0c54106500 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 msgr2=0x7f0c4003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 0x7f0c4003aba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c006fd0 tx=0x7f0c3c006e40 comp rx=0 tx=0).stop 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 msgr2=0x7f0c5407bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009fb0 tx=0x7f0c44011890 comp rx=0 tx=0).stop 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 shutdown_connections 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0c400386e0 0x7f0c4003aba0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 --2- 172.21.15.26:0/1480465146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c54106500 0x7f0c5407bb00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 >> 172.21.15.26:0/1480465146 conn(0x7f0c54101b40 msgr2=0x7f0c541026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.037 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 shutdown_connections 2024-03-24T10:34:57.038 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:56.309+0000 7f0c5a61a700 1 -- 172.21.15.26:0/1480465146 wait complete. 2024-03-24T10:34:57.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:57 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1480465146' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T10:34:57.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:57 smithi026 ceph-mon[29210]: mgrmap e9: smithi026.bbjcru(active, since 26s) 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "smithi026.bbjcru", 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T10:34:58.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e1456d700 1 Processor -- start 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e1456d700 1 -- start start 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e1456d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e1456d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c079e50 con 0x7f0e0c106fb0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e12309700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e12309700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47666/0 (socket says 172.21.15.26:47666) 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e12309700 1 -- 172.21.15.26:0/3782578672 learned_addr learned my addr 172.21.15.26:0/3782578672 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.503+0000 7f0e12309700 1 -- 172.21.15.26:0/3782578672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c1098e0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e12309700 1 --2- 172.21.15.26:0/3782578672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e080098a0 tx=0x7f0e08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dc8a0f14675a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e11307700 1 -- 172.21.15.26:0/3782578672 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e08004180 con 0x7f0e0c106fb0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e11307700 1 -- 172.21.15.26:0/3782578672 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e080042e0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e11307700 1 -- 172.21.15.26:0/3782578672 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e08011b10 con 0x7f0e0c106fb0 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e1456d700 1 -- 172.21.15.26:0/3782578672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 msgr2=0x7f0e0c1093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e1456d700 1 --2- 172.21.15.26:0/3782578672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e080098a0 tx=0x7f0e08009bb0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e1456d700 1 -- 172.21.15.26:0/3782578672 shutdown_connections 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e1456d700 1 --2- 172.21.15.26:0/3782578672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c1093a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.504+0000 7f0e1456d700 1 -- 172.21.15.26:0/3782578672 >> 172.21.15.26:0/3782578672 conn(0x7f0e0c100980 msgr2=0x7f0e0c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 -- 172.21.15.26:0/3782578672 shutdown_connections 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 -- 172.21.15.26:0/3782578672 wait complete. 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 Processor -- start 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 -- start start 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e1456d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c19a7a0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e12309700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e12309700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47680/0 (socket says 172.21.15.26:47680) 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.505+0000 7f0e12309700 1 -- 172.21.15.26:0/2351356629 learned_addr learned my addr 172.21.15.26:0/2351356629 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0e12309700 1 -- 172.21.15.26:0/2351356629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08009550 con 0x7f0e0c106fb0 2024-03-24T10:34:58.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0e12309700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e080117b0 tx=0x7f0e08011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e080038b0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e08003a10 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c19a9a0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e08024780 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.506+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c19ae40 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f0e080248e0 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c10a670 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0dff7fe700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 0x7f0df803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0e0804ca60 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0e11b08700 1 -- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 msgr2=0x7f0df803ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.507+0000 7f0e11b08700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 0x7f0df803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.512+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e08010380 con 0x7f0e0c106fb0 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.708+0000 7f0e11b08700 1 -- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 msgr2=0x7f0df803ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:34:58.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.708+0000 7f0e11b08700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 0x7f0df803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.772+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0e0c10a860 con 0x7f0e0c106fb0 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.773+0000 7f0dff7fe700 1 -- 172.21.15.26:0/2351356629 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f0e08019e40 con 0x7f0e0c106fb0 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 msgr2=0x7f0df803ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 0x7f0df803ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 msgr2=0x7f0e0c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e080117b0 tx=0x7f0e08011890 comp rx=0 tx=0).stop 2024-03-24T10:34:58.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 shutdown_connections 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7f0df8038770 0x7f0df803ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 --2- 172.21.15.26:0/2351356629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e0c106fb0 0x7f0e0c19a260 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 >> 172.21.15.26:0/2351356629 conn(0x7f0e0c100980 msgr2=0x7f0e0c101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 shutdown_connections 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:57.775+0000 7f0e1456d700 1 -- 172.21.15.26:0/2351356629 wait complete. 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-24T10:34:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 9... 2024-03-24T10:34:58.572 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:34:58 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2351356629' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: Active manager daemon smithi026.bbjcru restarted 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: Activating manager daemon smithi026.bbjcru 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: mgrmap e10: smithi026.bbjcru(active, starting, since 0.00316544s) 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:35:03.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi026.bbjcru", "id": "smithi026.bbjcru"}]: dispatch 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: Manager daemon smithi026.bbjcru is now available 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:03.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:03 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:04 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:35:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:04 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/trash_purge_schedule"}]: dispatch 2024-03-24T10:35:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:04 smithi026 ceph-mon[29210]: mgrmap e11: smithi026.bbjcru(active, since 1.00557s) 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2b24fc700 1 Processor -- start 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2b24fc700 1 -- start start 2024-03-24T10:35:05.007 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2b24fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2b24fc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac079dd0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47688/0 (socket says 172.21.15.26:47688) 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2abfff700 1 -- 172.21.15.26:0/307315658 learned_addr learned my addr 172.21.15.26:0/307315658 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.786+0000 7fa2abfff700 1 -- 172.21.15.26:0/307315658 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac079f10 con 0x7fa2ac106ff0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2abfff700 1 --2- 172.21.15.26:0/307315658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0098a0 tx=0x7fa29c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2279a89638702405 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2aaffd700 1 -- 172.21.15.26:0/307315658 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c004180 con 0x7fa2ac106ff0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2aaffd700 1 -- 172.21.15.26:0/307315658 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29c0042e0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2aaffd700 1 -- 172.21.15.26:0/307315658 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c011b10 con 0x7fa2ac106ff0 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2b24fc700 1 -- 172.21.15.26:0/307315658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 msgr2=0x7fa2ac1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/307315658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0098a0 tx=0x7fa29c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2b24fc700 1 -- 172.21.15.26:0/307315658 shutdown_connections 2024-03-24T10:35:05.008 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/307315658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac1093e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.787+0000 7fa2b24fc700 1 -- 172.21.15.26:0/307315658 >> 172.21.15.26:0/307315658 conn(0x7fa2ac100980 msgr2=0x7fa2ac102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 -- 172.21.15.26:0/307315658 shutdown_connections 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 -- 172.21.15.26:0/307315658 wait complete. 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 Processor -- start 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 -- start start 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2b24fc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac19a740 con 0x7fa2ac106ff0 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.788+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47694/0 (socket says 172.21.15.26:47694) 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2abfff700 1 -- 172.21.15.26:0/2858854498 learned_addr learned my addr 172.21.15.26:0/2858854498 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2abfff700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c009550 con 0x7fa2ac106ff0 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2abfff700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009fb0 tx=0x7fa29c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c0038b0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.009 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29c003a10 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac19a940 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c019750 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.789+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac19ade0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7fa29c0198b0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2a97fa700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29403b350 con 0x7fa294038780 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2ab7fe700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa29c04d8a0 con 0x7fa2ac106ff0 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.790+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:35:05.010 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.991+0000 7fa2ab7fe700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:58.991+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:59.392+0000 7fa2ab7fe700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:34:59.392+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:00.193+0000 7fa2ab7fe700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:00.193+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:01.794+0000 7fa2ab7fe700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/3234553684 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:01.794+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:03.262+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 10) v1 ==== 45328+0+0 (secure 0 0 0) 0x7fa29c01ac10 con 0x7fa2ac106ff0 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:03.262+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 msgr2=0x7fa29403ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:03.262+0000 7fa2a97fa700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.011 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.262+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7fa29c02c440 con 0x7fa2ac106ff0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.262+0000 7fa2a97fa700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 0x7fa29403e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.262+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29403b350 con 0x7fa29403bfd0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.262+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 0x7fa29403e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.263+0000 7fa2ab7fe700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 0x7fa29403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0003a10 tx=0x7fa2a0009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.264+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fa29403b350 con 0x7fa29403bfd0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa2ac10a640 con 0x7fa29403bfd0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2a97fa700 1 -- 172.21.15.26:0/2858854498 <== mgr.14164 v2:172.21.15.26:6800/1986399491 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa2ac10a640 con 0x7fa29403bfd0 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 msgr2=0x7fa29403e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 0x7fa29403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0003a10 tx=0x7fa2a0009270 comp rx=0 tx=0).stop 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 msgr2=0x7fa2ac19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.270+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009fb0 tx=0x7fa29c011890 comp rx=0 tx=0).stop 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 shutdown_connections 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2ac106ff0 0x7fa2ac19a200 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.012 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa29403bfd0 0x7fa29403e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 --2- 172.21.15.26:0/2858854498 >> [v2:172.21.15.26:6800/3234553684,v1:172.21.15.26:6801/3234553684] conn(0x7fa294038780 0x7fa29403ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 >> 172.21.15.26:0/2858854498 conn(0x7fa2ac100980 msgr2=0x7fa2ac1014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 shutdown_connections 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:04.271+0000 7fa2b24fc700 1 -- 172.21.15.26:0/2858854498 wait complete. 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 9 is available 2024-03-24T10:35:05.013 INFO:teuthology.orchestra.run.smithi026.stdout:Generating a dashboard self-signed certificate... 2024-03-24T10:35:05.616 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: [24/Mar/2024:10:35:04] ENGINE Bus STARTING 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: [24/Mar/2024:10:35:04] ENGINE Serving on http://172.21.15.26:8765 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: [24/Mar/2024:10:35:04] ENGINE Serving on https://172.21.15.26:7150 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: [24/Mar/2024:10:35:04] ENGINE Bus STARTED 2024-03-24T10:35:05.617 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:05 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:06.486 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T10:35:06.487 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.621+0000 7f1396bea700 1 Processor -- start 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.621+0000 7f1396bea700 1 -- start start 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.621+0000 7f1396bea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.621+0000 7f1396bea700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1390106f00 con 0x7f1390106510 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f1394986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f1394986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49674/0 (socket says 172.21.15.26:49674) 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f1394986700 1 -- 172.21.15.26:0/167926662 learned_addr learned my addr 172.21.15.26:0/167926662 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f1394986700 1 -- 172.21.15.26:0/167926662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13901076c0 con 0x7f1390106510 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f1394986700 1 --2- 172.21.15.26:0/167926662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f1384009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19df9124e1e63fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f138f7fe700 1 -- 172.21.15.26:0/167926662 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1384004180 con 0x7f1390106510 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f138f7fe700 1 -- 172.21.15.26:0/167926662 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13840042e0 con 0x7f1390106510 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.622+0000 7f138f7fe700 1 -- 172.21.15.26:0/167926662 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1384011b10 con 0x7f1390106510 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 -- 172.21.15.26:0/167926662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 msgr2=0x7f1390106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 --2- 172.21.15.26:0/167926662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f1384009bb0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 -- 172.21.15.26:0/167926662 shutdown_connections 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 --2- 172.21.15.26:0/167926662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f1390106930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 -- 172.21.15.26:0/167926662 >> 172.21.15.26:0/167926662 conn(0x7f1390101b70 msgr2=0x7f1390103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 -- 172.21.15.26:0/167926662 shutdown_connections 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.623+0000 7f1396bea700 1 -- 172.21.15.26:0/167926662 wait complete. 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1396bea700 1 Processor -- start 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1396bea700 1 -- start start 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1396bea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1396bea700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139019a7b0 con 0x7f1390106510 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1394986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1394986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49684/0 (socket says 172.21.15.26:49684) 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1394986700 1 -- 172.21.15.26:0/3844084045 learned_addr learned my addr 172.21.15.26:0/3844084045 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1394986700 1 -- 172.21.15.26:0/3844084045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384009550 con 0x7f1390106510 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.624+0000 7f1394986700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f13840117b0 tx=0x7f1384011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.625+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13840038b0 con 0x7f1390106510 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.625+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139019a9b0 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.625+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1384003a10 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.625+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13840246e0 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.625+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139019ae50 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7f138402d3f0 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f138dffb700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 0x7f138003ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f138404c870 con 0x7f1390106510 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f138ffff700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 0x7f138003ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13900623e0 con 0x7f1390106510 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.626+0000 7f138ffff700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 0x7f138003ab10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.631+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1384010c00 con 0x7f1390106510 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.868+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1390102d50 con 0x7f1380038650 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.933+0000 7f138dffb700 1 -- 172.21.15.26:0/3844084045 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1390102d50 con 0x7f1380038650 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 msgr2=0x7f138003ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 0x7f138003ab10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 msgr2=0x7f139019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f13840117b0 tx=0x7f1384011890 comp rx=0 tx=0).stop 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 shutdown_connections 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1380038650 0x7f138003ab10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 --2- 172.21.15.26:0/3844084045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1390106510 0x7f139019a270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 >> 172.21.15.26:0/3844084045 conn(0x7f1390101b70 msgr2=0x7f1390102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 shutdown_connections 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:05.936+0000 7f1396bea700 1 -- 172.21.15.26:0/3844084045 wait complete. 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial admin user... 2024-03-24T10:35:07.150 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:06 smithi026 ceph-mon[29210]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:07.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:07.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:07.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:07.151 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:06 smithi026 ceph-mon[29210]: mgrmap e12: smithi026.bbjcru(active, since 2s) 2024-03-24T10:35:08.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:07 smithi026 ceph-mon[29210]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:08.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:07 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$rEEhPh1etD2brk9UYGoaEOxU3F2a.EugkJ8eNC88sH1WiWNoUUZOa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711276507, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.919+0000 7f5db6767700 1 Processor -- start 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.919+0000 7f5db6767700 1 -- start start 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.919+0000 7f5db6767700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.919+0000 7f5db6767700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db007a820 con 0x7f5db007bde0 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49686/0 (socket says 172.21.15.26:49686) 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daffff700 1 -- 172.21.15.26:0/117650306 learned_addr learned my addr 172.21.15.26:0/117650306 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:08.618 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daffff700 1 -- 172.21.15.26:0/117650306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db007a960 con 0x7f5db007bde0 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daffff700 1 --2- 172.21.15.26:0/117650306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5da00098a0 tx=0x7f5da0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b06e28e5623a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daeffd700 1 -- 172.21.15.26:0/117650306 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da0004180 con 0x7f5db007bde0 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daeffd700 1 -- 172.21.15.26:0/117650306 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5da00042e0 con 0x7f5db007bde0 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.920+0000 7f5daeffd700 1 -- 172.21.15.26:0/117650306 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da0011b10 con 0x7f5db007bde0 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 -- 172.21.15.26:0/117650306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 msgr2=0x7f5db007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 --2- 172.21.15.26:0/117650306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5da00098a0 tx=0x7f5da0009bb0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 -- 172.21.15.26:0/117650306 shutdown_connections 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 --2- 172.21.15.26:0/117650306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db007a2e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 -- 172.21.15.26:0/117650306 >> 172.21.15.26:0/117650306 conn(0x7f5db0101b40 msgr2=0x7f5db0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 -- 172.21.15.26:0/117650306 shutdown_connections 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.921+0000 7f5db6767700 1 -- 172.21.15.26:0/117650306 wait complete. 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5db6767700 1 Processor -- start 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5db6767700 1 -- start start 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5db6767700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5db6767700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db019c930 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5daffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5daffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49696/0 (socket says 172.21.15.26:49696) 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5daffff700 1 -- 172.21.15.26:0/4156667588 learned_addr learned my addr 172.21.15.26:0/4156667588 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.922+0000 7f5daffff700 1 -- 172.21.15.26:0/4156667588 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da0009550 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5daffff700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5da00117b0 tx=0x7f5da0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da00038b0 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db019cb30 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5da0003a10 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da0024780 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.923+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db019cfd0 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.924+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f5da00248e0 con 0x7f5db007bde0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.924+0000 7f5dad7fa700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 0x7f5d9803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.924+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5da004ca60 con 0x7f5db007bde0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.924+0000 7f5daf7fe700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 0x7f5d9803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.924+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db00623e0 con 0x7f5db007bde0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.925+0000 7f5daf7fe700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 0x7f5d9803ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5da4006fd0 tx=0x7f5da4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:06.929+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5da0024be0 con 0x7f5db007bde0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.170+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5db0062200 con 0x7f5d98038770 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.419+0000 7f5dad7fa700 1 -- 172.21.15.26:0/4156667588 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f5db0062200 con 0x7f5d98038770 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 msgr2=0x7f5d9803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 0x7f5d9803ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5da4006fd0 tx=0x7f5da4006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 msgr2=0x7f5db019c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5da00117b0 tx=0x7f5da0011890 comp rx=0 tx=0).stop 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 shutdown_connections 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f5d98038770 0x7f5d9803ac30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 --2- 172.21.15.26:0/4156667588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5db007bde0 0x7f5db019c3f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 >> 172.21.15.26:0/4156667588 conn(0x7f5db0101b40 msgr2=0x7f5db01026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.622 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 shutdown_connections 2024-03-24T10:35:08.622 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:07.422+0000 7f5db6767700 1 -- 172.21.15.26:0/4156667588 wait complete. 2024-03-24T10:35:08.622 INFO:teuthology.orchestra.run.smithi026.stdout:Fetching dashboard port number... 2024-03-24T10:35:09.571 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:09 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/4143457687' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa272998700 1 Processor -- start 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa272998700 1 -- start start 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa272998700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa272998700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c106f40 con 0x7fa26c106550 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.857 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49706/0 (socket says 172.21.15.26:49706) 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.060+0000 7fa26bfff700 1 -- 172.21.15.26:0/3097312632 learned_addr learned my addr 172.21.15.26:0/3097312632 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.061+0000 7fa26bfff700 1 -- 172.21.15.26:0/3097312632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c1076b0 con 0x7fa26c106550 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.061+0000 7fa26bfff700 1 --2- 172.21.15.26:0/3097312632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098a0 tx=0x7fa25c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2443bf43a6757c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.061+0000 7fa26affd700 1 -- 172.21.15.26:0/3097312632 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa25c004180 con 0x7fa26c106550 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.061+0000 7fa26affd700 1 -- 172.21.15.26:0/3097312632 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa25c0042e0 con 0x7fa26c106550 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.061+0000 7fa26affd700 1 -- 172.21.15.26:0/3097312632 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa25c011b10 con 0x7fa26c106550 2024-03-24T10:35:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- 172.21.15.26:0/3097312632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 msgr2=0x7fa26c106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 --2- 172.21.15.26:0/3097312632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098a0 tx=0x7fa25c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- 172.21.15.26:0/3097312632 shutdown_connections 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 --2- 172.21.15.26:0/3097312632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c106970 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- 172.21.15.26:0/3097312632 >> 172.21.15.26:0/3097312632 conn(0x7fa26c101b10 msgr2=0x7fa26c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- 172.21.15.26:0/3097312632 shutdown_connections 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- 172.21.15.26:0/3097312632 wait complete. 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 Processor -- start 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.062+0000 7fa272998700 1 -- start start 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa272998700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa272998700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c07c040 con 0x7fa26c106550 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49714/0 (socket says 172.21.15.26:49714) 2024-03-24T10:35:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa26bfff700 1 -- 172.21.15.26:0/4143457687 learned_addr learned my addr 172.21.15.26:0/4143457687 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa26bfff700 1 -- 172.21.15.26:0/4143457687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c009550 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa26bfff700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0117b0 tx=0x7fa25c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa25c0038b0 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c07a280 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa25c003a10 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa25c0246e0 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.063+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c07a720 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.064+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7fa25c02d3f0 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.064+0000 7fa2697fa700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 0x7fa25803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.064+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c0623e0 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.064+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa25c04c9a0 con 0x7fa26c106550 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.064+0000 7fa26b7fe700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 0x7fa25803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.065+0000 7fa26b7fe700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 0x7fa25803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa254006fd0 tx=0x7fa254006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.860 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.069+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa25c02d740 con 0x7fa26c106550 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.271+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa26c10a8b0 con 0x7fa26c106550 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.272+0000 7fa2697fa700 1 -- 172.21.15.26:0/4143457687 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa25c0107f0 con 0x7fa26c106550 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 msgr2=0x7fa25803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 0x7fa25803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa254006fd0 tx=0x7fa254006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 msgr2=0x7fa26c07bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0117b0 tx=0x7fa25c011890 comp rx=0 tx=0).stop 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 shutdown_connections 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fa258038770 0x7fa25803ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 --2- 172.21.15.26:0/4143457687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa26c106550 0x7fa26c07bb00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 >> 172.21.15.26:0/4143457687 conn(0x7fa26c101b10 msgr2=0x7fa26c102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 shutdown_connections 2024-03-24T10:35:09.861 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:09.274+0000 7fa272998700 1 -- 172.21.15.26:0/4143457687 wait complete. 2024-03-24T10:35:09.866 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:35:09.866 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-24T10:35:09.870 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:35:09.870 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-24T10:35:09.870 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-24T10:35:09.870 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T10:35:09.872 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph Dashboard is now available at: 2024-03-24T10:35:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: URL: https://smithi026.front.sepia.ceph.com:8443/ 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: User: admin 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: Password: t9lrf2ruyo 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout:Saving cluster configuration to /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config directory 2024-03-24T10:35:09.873 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling autotune for osd_memory_target 2024-03-24T10:35:10.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:10 smithi026 ceph-mon[29210]: mgrmap e13: smithi026.bbjcru(active, since 6s) 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.322+0000 7fb58c8d3700 1 Processor -- start 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.322+0000 7fb58c8d3700 1 -- start start 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.322+0000 7fb58c8d3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.322+0000 7fb58c8d3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58807a8b0 con 0x7fb58807be30 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49728/0 (socket says 172.21.15.26:49728) 2024-03-24T10:35:11.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58659c700 1 -- 172.21.15.26:0/2445101167 learned_addr learned my addr 172.21.15.26:0/2445101167 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58659c700 1 -- 172.21.15.26:0/2445101167 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58807a9f0 con 0x7fb58807be30 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58659c700 1 --2- 172.21.15.26:0/2445101167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb580009e80 tx=0x7fb58000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2c18ee09830a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58559a700 1 -- 172.21.15.26:0/2445101167 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58000e070 con 0x7fb58807be30 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58559a700 1 -- 172.21.15.26:0/2445101167 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb58000b8c0 con 0x7fb58807be30 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.323+0000 7fb58559a700 1 -- 172.21.15.26:0/2445101167 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5800044d0 con 0x7fb58807be30 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 -- 172.21.15.26:0/2445101167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 msgr2=0x7fb58807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/2445101167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb580009e80 tx=0x7fb58000b330 comp rx=0 tx=0).stop 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 -- 172.21.15.26:0/2445101167 shutdown_connections 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/2445101167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58807a2e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 -- 172.21.15.26:0/2445101167 >> 172.21.15.26:0/2445101167 conn(0x7fb588101b10 msgr2=0x7fb588103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 -- 172.21.15.26:0/2445101167 shutdown_connections 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.324+0000 7fb58c8d3700 1 -- 172.21.15.26:0/2445101167 wait complete. 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58c8d3700 1 Processor -- start 2024-03-24T10:35:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58c8d3700 1 -- start start 2024-03-24T10:35:11.046 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58c8d3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.046 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58c8d3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58819c910 con 0x7fb58807be30 2024-03-24T10:35:11.046 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.046 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49742/0 (socket says 172.21.15.26:49742) 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58659c700 1 -- 172.21.15.26:0/1624371485 learned_addr learned my addr 172.21.15.26:0/1624371485 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58659c700 1 -- 172.21.15.26:0/1624371485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580009b30 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.325+0000 7fb58659c700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb5800072a0 tx=0x7fb58000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58000e040 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb580003dc0 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58819cb10 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb580012750 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58819d010 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.326+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7fb580019040 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.327+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58810c850 con 0x7fb58807be30 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.327+0000 7fb57f7fe700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 0x7fb57403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.327+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb58004c5c0 con 0x7fb58807be30 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.327+0000 7fb585d9b700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 0x7fb57403acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.327+0000 7fb585d9b700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 0x7fb57403acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.331+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb580011750 con 0x7fb58807be30 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.530+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb58810ca70 con 0x7fb58807be30 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.530+0000 7fb57f7fe700 1 -- 172.21.15.26:0/1624371485 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb58001f070 con 0x7fb58807be30 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 msgr2=0x7fb57403acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 0x7fb57403acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 msgr2=0x7fb58819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb5800072a0 tx=0x7fb58000be00 comp rx=0 tx=0).stop 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 shutdown_connections 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fb574038810 0x7fb57403acd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.048 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 --2- 172.21.15.26:0/1624371485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb58807be30 0x7fb58819c3d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.049 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 >> 172.21.15.26:0/1624371485 conn(0x7fb588101b10 msgr2=0x7fb588102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.049 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 shutdown_connections 2024-03-24T10:35:11.049 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:10.532+0000 7fb58c8d3700 1 -- 172.21.15.26:0/1624371485 wait complete. 2024-03-24T10:35:12.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f767616c700 1 Processor -- start 2024-03-24T10:35:12.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f767616c700 1 -- start start 2024-03-24T10:35:12.410 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f767616c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f767616c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670079e90 con 0x7f7670106f30 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49752/0 (socket says 172.21.15.26:49752) 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.506+0000 7f766f7fe700 1 -- 172.21.15.26:0/2293140902 learned_addr learned my addr 172.21.15.26:0/2293140902 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.507+0000 7f766f7fe700 1 -- 172.21.15.26:0/2293140902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670109860 con 0x7f7670106f30 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.507+0000 7f766f7fe700 1 --2- 172.21.15.26:0/2293140902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f76580098a0 tx=0x7f7658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d3c611959607f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.507+0000 7f766e7fc700 1 -- 172.21.15.26:0/2293140902 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7658004180 con 0x7f7670106f30 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.507+0000 7f766e7fc700 1 -- 172.21.15.26:0/2293140902 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76580042e0 con 0x7f7670106f30 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.507+0000 7f766e7fc700 1 -- 172.21.15.26:0/2293140902 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7658011b10 con 0x7f7670106f30 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 -- 172.21.15.26:0/2293140902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 msgr2=0x7f7670109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 --2- 172.21.15.26:0/2293140902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f76580098a0 tx=0x7f7658009bb0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 -- 172.21.15.26:0/2293140902 shutdown_connections 2024-03-24T10:35:12.411 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 --2- 172.21.15.26:0/2293140902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f7670109320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 -- 172.21.15.26:0/2293140902 >> 172.21.15.26:0/2293140902 conn(0x7f7670100980 msgr2=0x7f7670102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 -- 172.21.15.26:0/2293140902 shutdown_connections 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 -- 172.21.15.26:0/2293140902 wait complete. 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.508+0000 7f767616c700 1 Processor -- start 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f767616c700 1 -- start start 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f767616c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f767616c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767019a750 con 0x7f7670106f30 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49754/0 (socket says 172.21.15.26:49754) 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f766f7fe700 1 -- 172.21.15.26:0/1632601432 learned_addr learned my addr 172.21.15.26:0/1632601432 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f766f7fe700 1 -- 172.21.15.26:0/1632601432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009550 con 0x7f7670106f30 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.509+0000 7f766f7fe700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7658009fb0 tx=0x7f7658011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.412 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76580038b0 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767019a950 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7658003a10 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7658019750 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767019adf0 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.510+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f76580199c0 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.511+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767010a650 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.511+0000 7f766cff9700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 0x7f765c03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.511+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f765804cb70 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.511+0000 7f766effd700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 0x7f765c03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.511+0000 7f766effd700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 0x7f765c03ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fd0 tx=0x7f7660006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.515+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7658010380 con 0x7f7670106f30 2024-03-24T10:35:12.413 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.827+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f767010a840 con 0x7f7670106f30 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.830+0000 7f766cff9700 1 -- 172.21.15.26:0/1632601432 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f7658022b00 con 0x7f7670106f30 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 msgr2=0x7f765c03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 0x7f765c03ac90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fd0 tx=0x7f7660006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 msgr2=0x7f767019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7658009fb0 tx=0x7f7658011890 comp rx=0 tx=0).stop 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 shutdown_connections 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f765c0387d0 0x7f765c03ac90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 --2- 172.21.15.26:0/1632601432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7670106f30 0x7f767019a210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 >> 172.21.15.26:0/1632601432 conn(0x7f7670100980 msgr2=0x7f7670101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 shutdown_connections 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-24T10:35:11.832+0000 7f767616c700 1 -- 172.21.15.26:0/1632601432 wait complete. 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T10:35:12.414 INFO:teuthology.orchestra.run.smithi026.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: ceph telemetry on 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout:For more information see: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T10:35:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:12.416 INFO:teuthology.orchestra.run.smithi026.stdout:Bootstrap complete. 2024-03-24T10:35:12.446 INFO:tasks.cephadm:Fetching config... 2024-03-24T10:35:12.446 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:35:12.446 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T10:35:12.462 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T10:35:12.462 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:35:12.462 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T10:35:12.518 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T10:35:12.518 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:35:12.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/keyring of=/dev/stdout 2024-03-24T10:35:12.585 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T10:35:12.585 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:35:12.585 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T10:35:12.642 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T10:35:12.642 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdaXINS/3gAlWcLH0Wu/0967EWjrL4Kmf1YI7HEkOHAVCU23OdjS45cDY/CaPFGSyYcqZRUXGUSJsPBC+3mjBoeDNIacj15vsdiY5UCYbZahOZyPqOilkK0N5qMXiW6BO0G4NLKQvP6mmzUeDT+PkfBhwKyDcJLs0WwXRSYsWmdxHcdT9ZOFnRL3tNv5J8/eA4SmbLhXEOltw+YOxWVinwFMp4V4s56Aet8Ggu2OrAXopMoTFK4GdQ81Hm665PHeXWTwIOQa8x1PIMvrpLMXyr1hkBt9IPrWEb+OP+SEDJyjdTMbQBOTKxFr5OjLSiJMOOgS4JNWboCizptOUhQB7dJwQRBEg8EnVfRRwLi63ZZr5DVvujdvJZGbx3s0cwvFZhLM0zPnmOpPfC84TJHbGyJEtBst61Rlnw34P4BcYxgrRBqnjILkJ9222g1eM5e30E5ZXwdw+qHmx3QJgGElVPLVnXoyuEJXbkVfcH3b4FUw3K1qHk9aa5wtknglm6Exs= ceph-e07c400a-e9c9-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T10:35:12.720 INFO:teuthology.orchestra.run.smithi026.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdaXINS/3gAlWcLH0Wu/0967EWjrL4Kmf1YI7HEkOHAVCU23OdjS45cDY/CaPFGSyYcqZRUXGUSJsPBC+3mjBoeDNIacj15vsdiY5UCYbZahOZyPqOilkK0N5qMXiW6BO0G4NLKQvP6mmzUeDT+PkfBhwKyDcJLs0WwXRSYsWmdxHcdT9ZOFnRL3tNv5J8/eA4SmbLhXEOltw+YOxWVinwFMp4V4s56Aet8Ggu2OrAXopMoTFK4GdQ81Hm665PHeXWTwIOQa8x1PIMvrpLMXyr1hkBt9IPrWEb+OP+SEDJyjdTMbQBOTKxFr5OjLSiJMOOgS4JNWboCizptOUhQB7dJwQRBEg8EnVfRRwLi63ZZr5DVvujdvJZGbx3s0cwvFZhLM0zPnmOpPfC84TJHbGyJEtBst61Rlnw34P4BcYxgrRBqnjILkJ9222g1eM5e30E5ZXwdw+qHmx3QJgGElVPLVnXoyuEJXbkVfcH3b4FUw3K1qHk9aa5wtknglm6Exs= ceph-e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:35:12.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdaXINS/3gAlWcLH0Wu/0967EWjrL4Kmf1YI7HEkOHAVCU23OdjS45cDY/CaPFGSyYcqZRUXGUSJsPBC+3mjBoeDNIacj15vsdiY5UCYbZahOZyPqOilkK0N5qMXiW6BO0G4NLKQvP6mmzUeDT+PkfBhwKyDcJLs0WwXRSYsWmdxHcdT9ZOFnRL3tNv5J8/eA4SmbLhXEOltw+YOxWVinwFMp4V4s56Aet8Ggu2OrAXopMoTFK4GdQ81Hm665PHeXWTwIOQa8x1PIMvrpLMXyr1hkBt9IPrWEb+OP+SEDJyjdTMbQBOTKxFr5OjLSiJMOOgS4JNWboCizptOUhQB7dJwQRBEg8EnVfRRwLi63ZZr5DVvujdvJZGbx3s0cwvFZhLM0zPnmOpPfC84TJHbGyJEtBst61Rlnw34P4BcYxgrRBqnjILkJ9222g1eM5e30E5ZXwdw+qHmx3QJgGElVPLVnXoyuEJXbkVfcH3b4FUw3K1qHk9aa5wtknglm6Exs= ceph-e07c400a-e9c9-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T10:35:12.781 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdaXINS/3gAlWcLH0Wu/0967EWjrL4Kmf1YI7HEkOHAVCU23OdjS45cDY/CaPFGSyYcqZRUXGUSJsPBC+3mjBoeDNIacj15vsdiY5UCYbZahOZyPqOilkK0N5qMXiW6BO0G4NLKQvP6mmzUeDT+PkfBhwKyDcJLs0WwXRSYsWmdxHcdT9ZOFnRL3tNv5J8/eA4SmbLhXEOltw+YOxWVinwFMp4V4s56Aet8Ggu2OrAXopMoTFK4GdQ81Hm665PHeXWTwIOQa8x1PIMvrpLMXyr1hkBt9IPrWEb+OP+SEDJyjdTMbQBOTKxFr5OjLSiJMOOgS4JNWboCizptOUhQB7dJwQRBEg8EnVfRRwLi63ZZr5DVvujdvJZGbx3s0cwvFZhLM0zPnmOpPfC84TJHbGyJEtBst61Rlnw34P4BcYxgrRBqnjILkJ9222g1eM5e30E5ZXwdw+qHmx3QJgGElVPLVnXoyuEJXbkVfcH3b4FUw3K1qHk9aa5wtknglm6Exs= ceph-e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:35:12.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T10:35:12.855 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:12 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1632601432' entity='client.admin' 2024-03-24T10:35:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:35:15.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 -- 172.21.15.26:0/170713918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 msgr2=0x7f4eec0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:15.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 --2- 172.21.15.26:0/170713918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec0ffa60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009910 tx=0x7f4edc009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:15.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 -- 172.21.15.26:0/170713918 shutdown_connections 2024-03-24T10:35:15.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 --2- 172.21.15.26:0/170713918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec0ffa60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:15.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 -- 172.21.15.26:0/170713918 >> 172.21.15.26:0/170713918 conn(0x7f4eec0fabe0 msgr2=0x7f4eec0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:15.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 -- 172.21.15.26:0/170713918 shutdown_connections 2024-03-24T10:35:15.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 -- 172.21.15.26:0/170713918 wait complete. 2024-03-24T10:35:15.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.248+0000 7f4ef201c700 1 Processor -- start 2024-03-24T10:35:15.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4ef201c700 1 -- start start 2024-03-24T10:35:15.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4ef201c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:15.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4ef201c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec196100 con 0x7f4eec0ff640 2024-03-24T10:35:15.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:15.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55188/0 (socket says 172.21.15.26:55188) 2024-03-24T10:35:15.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4eeb7fe700 1 -- 172.21.15.26:0/2653301675 learned_addr learned my addr 172.21.15.26:0/2653301675 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:15.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4eeb7fe700 1 -- 172.21.15.26:0/2653301675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0095f0 con 0x7f4eec0ff640 2024-03-24T10:35:15.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.249+0000 7f4eeb7fe700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4edc006010 tx=0x7f4edc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:15.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4edc01d070 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec196300 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4edc005dc0 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4edc022e90 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec1967a0 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.250+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f4edc00f4f0 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.251+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec18f6f0 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.251+0000 7f4ee8ff9700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 0x7f4ed403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.251+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4edc04cce0 con 0x7f4eec0ff640 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.251+0000 7f4eeaffd700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 0x7f4ed403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:15.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.251+0000 7f4eeaffd700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 0x7f4ed403ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:15.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.256+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4edc02a950 con 0x7f4eec0ff640 2024-03-24T10:35:15.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.458+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4eec0623e0 con 0x7f4eec0ff640 2024-03-24T10:35:15.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.461+0000 7f4ee8ff9700 1 -- 172.21.15.26:0/2653301675 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4edc027020 con 0x7f4eec0ff640 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 msgr2=0x7f4ed403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 0x7f4ed403ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 msgr2=0x7f4eec195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4edc006010 tx=0x7f4edc009f80 comp rx=0 tx=0).stop 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 shutdown_connections 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4ed4038760 0x7f4ed403ac20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 --2- 172.21.15.26:0/2653301675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4eec0ff640 0x7f4eec195bc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 >> 172.21.15.26:0/2653301675 conn(0x7f4eec0fabe0 msgr2=0x7f4eec068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 shutdown_connections 2024-03-24T10:35:15.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:15.464+0000 7f4ef201c700 1 -- 172.21.15.26:0/2653301675 wait complete. 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi026", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi026", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:15 smithi026 ceph-mon[29210]: Deploying daemon ceph-exporter.smithi026 on smithi026 2024-03-24T10:35:16.323 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T10:35:16.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T10:35:16.537 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:35:16.571 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:16 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2653301675' entity='client.admin' 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- 172.21.15.26:0/3242196770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 msgr2=0x7fc0a81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 --2- 172.21.15.26:0/3242196770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a81035c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009910 tx=0x7fc09c009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- 172.21.15.26:0/3242196770 shutdown_connections 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 --2- 172.21.15.26:0/3242196770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a81035c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- 172.21.15.26:0/3242196770 >> 172.21.15.26:0/3242196770 conn(0x7fc0a80fabe0 msgr2=0x7fc0a80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- 172.21.15.26:0/3242196770 shutdown_connections 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- 172.21.15.26:0/3242196770 wait complete. 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 Processor -- start 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- start start 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.737+0000 7fc0aed47700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8100f40 con 0x7fc0a81011d0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0acae3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0acae3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55214/0 (socket says 172.21.15.26:55214) 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0acae3700 1 -- 172.21.15.26:0/4280202824 learned_addr learned my addr 172.21.15.26:0/4280202824 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0acae3700 1 -- 172.21.15.26:0/4280202824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0095f0 con 0x7fc0a81011d0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0acae3700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc09c006010 tx=0x7fc09c0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c01d070 con 0x7fc0a81011d0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a80ff180 con 0x7fc0a81011d0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc09c00be20 con 0x7fc0a81011d0 2024-03-24T10:35:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c00f460 con 0x7fc0a81011d0 2024-03-24T10:35:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.738+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a80ff620 con 0x7fc0a81011d0 2024-03-24T10:35:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.739+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7fc09c00f6d0 con 0x7fc0a81011d0 2024-03-24T10:35:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.739+0000 7fc0a5ffb700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 0x7fc09803ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.739+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc09c04da50 con 0x7fc0a81011d0 2024-03-24T10:35:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.740+0000 7fc0a7fff700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 0x7fc09803ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.740+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a8100030 con 0x7fc0a81011d0 2024-03-24T10:35:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.740+0000 7fc0a7fff700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 0x7fc09803ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc094006fd0 tx=0x7fc094006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.745+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc09c027070 con 0x7fc0a81011d0 2024-03-24T10:35:19.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.966+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0a818f280 con 0x7fc098038820 2024-03-24T10:35:19.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.970+0000 7fc0a5ffb700 1 -- 172.21.15.26:0/4280202824 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc0a818f280 con 0x7fc098038820 2024-03-24T10:35:19.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 msgr2=0x7fc09803ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:19.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 0x7fc09803ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc094006fd0 tx=0x7fc094006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 msgr2=0x7fc0a8100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc09c006010 tx=0x7fc09c0050b0 comp rx=0 tx=0).stop 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 shutdown_connections 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fc098038820 0x7fc09803ace0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 --2- 172.21.15.26:0/4280202824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0a81011d0 0x7fc0a8100a00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 >> 172.21.15.26:0/4280202824 conn(0x7fc0a80fabe0 msgr2=0x7fc0a80fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 shutdown_connections 2024-03-24T10:35:19.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:19.972+0000 7fc0aed47700 1 -- 172.21.15.26:0/4280202824 wait complete. 2024-03-24T10:35:20.590 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi031 2024-03-24T10:35:20.590 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:35:20.590 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/etc/ceph/ceph.conf 2024-03-24T10:35:20.608 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:35:20.609 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:35:20.666 INFO:tasks.cephadm:Adding host smithi031 to orchestrator... 2024-03-24T10:35:20.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch host add smithi031 2024-03-24T10:35:21.137 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:35:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:20 smithi026 ceph-mon[29210]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:20 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:22.225 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:22.226 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:22 smithi026 ceph-mon[29210]: Deploying daemon crash.smithi026 on smithi026 2024-03-24T10:35:22.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.984+0000 7f3d9bb64700 1 -- 172.21.15.26:0/463848098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 msgr2=0x7f3d94100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.984+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/463848098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94100b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009910 tx=0x7f3d90009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.984+0000 7f3d9bb64700 1 -- 172.21.15.26:0/463848098 shutdown_connections 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.984+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/463848098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94100b70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.984+0000 7f3d9bb64700 1 -- 172.21.15.26:0/463848098 >> 172.21.15.26:0/463848098 conn(0x7f3d940fbcf0 msgr2=0x7f3d940fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 -- 172.21.15.26:0/463848098 shutdown_connections 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 -- 172.21.15.26:0/463848098 wait complete. 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 Processor -- start 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 -- start start 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:22.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d9bb64700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94196140 con 0x7f3d94100750 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d99900700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d99900700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47168/0 (socket says 172.21.15.26:47168) 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.985+0000 7f3d99900700 1 -- 172.21.15.26:0/3781325150 learned_addr learned my addr 172.21.15.26:0/3781325150 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d99900700 1 -- 172.21.15.26:0/3781325150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900095f0 con 0x7f3d94100750 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d99900700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d90006010 tx=0x7f3d90009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d9001d070 con 0x7f3d94100750 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d94196340 con 0x7f3d94100750 2024-03-24T10:35:22.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d90005dc0 con 0x7f3d94100750 2024-03-24T10:35:22.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d90022e90 con 0x7f3d94100750 2024-03-24T10:35:22.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.986+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d941967e0 con 0x7f3d94100750 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.987+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f3d90022470 con 0x7f3d94100750 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.987+0000 7f3d8affd700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 0x7f3d8003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.987+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9418f580 con 0x7f3d94100750 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.987+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d9004c820 con 0x7f3d94100750 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.987+0000 7f3d990ff700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 0x7f3d8003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.988+0000 7f3d990ff700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 0x7f3d8003ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:22.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:22.992+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3d9000b960 con 0x7f3d94100750 2024-03-24T10:35:23.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:23.200+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d940611f0 con 0x7f3d80038760 2024-03-24T10:35:24.573 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:24 smithi026 ceph-mon[29210]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:24.574 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:24 smithi026 ceph-mon[29210]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:24.574 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:24 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:25.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:25 smithi026 ceph-mon[29210]: Deploying cephadm binary to smithi031 2024-03-24T10:35:25.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:25 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:25.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:25 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.964+0000 7f3d8affd700 1 -- 172.21.15.26:0/3781325150 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f3d940611f0 con 0x7f3d80038760 2024-03-24T10:35:25.969 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi031' with addr '172.21.15.31' 2024-03-24T10:35:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 msgr2=0x7f3d8003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 0x7f3d8003ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 msgr2=0x7f3d94195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d90006010 tx=0x7f3d90009f80 comp rx=0 tx=0).stop 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 shutdown_connections 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f3d80038760 0x7f3d8003ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 --2- 172.21.15.26:0/3781325150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d94100750 0x7f3d94195c00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 >> 172.21.15.26:0/3781325150 conn(0x7f3d940fbcf0 msgr2=0x7f3d940fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 shutdown_connections 2024-03-24T10:35:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:25.967+0000 7f3d9bb64700 1 -- 172.21.15.26:0/3781325150 wait complete. 2024-03-24T10:35:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:26 smithi026 ceph-mon[29210]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:26 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:26 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:26 smithi026 ceph-mon[29210]: Deploying daemon node-exporter.smithi026 on smithi026 2024-03-24T10:35:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:26 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:26.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T10:35:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:35:27.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:27 smithi026 ceph-mon[29210]: Added host smithi031 2024-03-24T10:35:28.667 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:28 smithi026 ceph-mon[29210]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:28.668 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:28 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:29.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 -- 172.21.15.26:0/1947036488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 msgr2=0x7f88640ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 --2- 172.21.15.26:0/1947036488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f88640ffa60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8854009910 tx=0x7f8854009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 -- 172.21.15.26:0/1947036488 shutdown_connections 2024-03-24T10:35:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 --2- 172.21.15.26:0/1947036488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f88640ffa60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 -- 172.21.15.26:0/1947036488 >> 172.21.15.26:0/1947036488 conn(0x7f88640fabe0 msgr2=0x7f88640fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 -- 172.21.15.26:0/1947036488 shutdown_connections 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.406+0000 7f886c307700 1 -- 172.21.15.26:0/1947036488 wait complete. 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886c307700 1 Processor -- start 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886c307700 1 -- start start 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886c307700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886c307700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8864196130 con 0x7f88640ff640 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886a0a3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886a0a3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47190/0 (socket says 172.21.15.26:47190) 2024-03-24T10:35:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886a0a3700 1 -- 172.21.15.26:0/3996345973 learned_addr learned my addr 172.21.15.26:0/3996345973 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886a0a3700 1 -- 172.21.15.26:0/3996345973 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88540095f0 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f886a0a3700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8854006010 tx=0x7f8854009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.407+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f885401d070 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8854005dc0 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8864196330 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8854022e90 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88641967d0 con 0x7f88640ff640 2024-03-24T10:35:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f885400f4b0 con 0x7f88640ff640 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.408+0000 7f885b7fe700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 0x7f885003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.409+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f885404c760 con 0x7f88640ff640 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.409+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8864105480 con 0x7f88640ff640 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.409+0000 7f88698a2700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 0x7f885003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.409+0000 7f88698a2700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 0x7f885003ac80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.414+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8854027070 con 0x7f88640ff640 2024-03-24T10:35:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.621+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8864068600 con 0x7f88500387c0 2024-03-24T10:35:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.623+0000 7f885b7fe700 1 -- 172.21.15.26:0/3996345973 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f8864068600 con 0x7f88500387c0 2024-03-24T10:35:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:35:29.623 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}] 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.624+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 msgr2=0x7f885003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.624+0000 7f886c307700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 0x7f885003ac80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 msgr2=0x7f8864195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8854006010 tx=0x7f8854009f80 comp rx=0 tx=0).stop 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 shutdown_connections 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88500387c0 0x7f885003ac80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 --2- 172.21.15.26:0/3996345973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88640ff640 0x7f8864195bf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 >> 172.21.15.26:0/3996345973 conn(0x7f88640fabe0 msgr2=0x7f8864067ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 shutdown_connections 2024-03-24T10:35:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:29.625+0000 7f886c307700 1 -- 172.21.15.26:0/3996345973 wait complete. 2024-03-24T10:35:30.403 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:30 smithi026 ceph-mon[29210]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:30.403 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:30 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:30.403 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:30 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:30.403 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:30 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:30.403 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:30 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:30.749 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T10:35:30.750 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:35:31.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:31 smithi026 ceph-mon[29210]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:35:31.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:31 smithi026 ceph-mon[29210]: Deploying daemon alertmanager.smithi026 on smithi026 2024-03-24T10:35:32.691 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:32 smithi026 ceph-mon[29210]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 -- 172.21.15.26:0/945442769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 msgr2=0x7fccd80ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 --2- 172.21.15.26:0/945442769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd80ffa60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009910 tx=0x7fccd4009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 -- 172.21.15.26:0/945442769 shutdown_connections 2024-03-24T10:35:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 --2- 172.21.15.26:0/945442769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd80ffa60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 -- 172.21.15.26:0/945442769 >> 172.21.15.26:0/945442769 conn(0x7fccd80fabe0 msgr2=0x7fccd80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 -- 172.21.15.26:0/945442769 shutdown_connections 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.172+0000 7fccdfe84700 1 -- 172.21.15.26:0/945442769 wait complete. 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccdfe84700 1 Processor -- start 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccdfe84700 1 -- start start 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccdfe84700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccdfe84700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8198330 con 0x7fccd80ff640 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccddc20700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccddc20700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44914/0 (socket says 172.21.15.26:44914) 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccddc20700 1 -- 172.21.15.26:0/567666233 learned_addr learned my addr 172.21.15.26:0/567666233 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccddc20700 1 -- 172.21.15.26:0/567666233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40095f0 con 0x7fccd80ff640 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.173+0000 7fccddc20700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006010 tx=0x7fccd40050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd401d070 con 0x7fccd80ff640 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccd400be20 con 0x7fccd80ff640 2024-03-24T10:35:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd8198530 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd4022de0 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd81989d0 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.174+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7fccd400f4b0 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.175+0000 7fccceffd700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 0x7fccc403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.175+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fccd404ca10 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.175+0000 7fccdd41f700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 0x7fccc403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.175+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd8191880 con 0x7fccd80ff640 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.175+0000 7fccdd41f700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 0x7fccc403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006fd0 tx=0x7fccc8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:33.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.180+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fccd4027070 con 0x7fccd80ff640 2024-03-24T10:35:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:33.393+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fccd8105da0 con 0x7fccd80ff640 2024-03-24T10:35:34.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.338+0000 7fccceffd700 1 -- 172.21.15.26:0/567666233 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fccd40209c0 con 0x7fccd80ff640 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 msgr2=0x7fccc403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 0x7fccc403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006fd0 tx=0x7fccc8006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 msgr2=0x7fccd8197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006010 tx=0x7fccd40050b0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 shutdown_connections 2024-03-24T10:35:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7fccc40387b0 0x7fccc403ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 --2- 172.21.15.26:0/567666233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fccd80ff640 0x7fccd8197df0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 >> 172.21.15.26:0/567666233 conn(0x7fccd80fabe0 msgr2=0x7fccd8068470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 shutdown_connections 2024-03-24T10:35:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:35:34.340+0000 7fccdfe84700 1 -- 172.21.15.26:0/567666233 wait complete. 2024-03-24T10:35:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-03-24T10:35:34.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:34 smithi026 ceph-mon[29210]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:34.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:34 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:34.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:34 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/567666233' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T10:35:35.637 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:35 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/567666233' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T10:35:35.637 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:35 smithi026 ceph-mon[29210]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:35:35.662 INFO:tasks.cephadm:Adding mon.smithi026 on smithi026 2024-03-24T10:35:35.662 INFO:tasks.cephadm:Adding mon.smithi031 on smithi031 2024-03-24T10:35:35.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi026:172.21.15.26=smithi026;smithi031:172.21.15.31=smithi031' 2024-03-24T10:35:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:35.912 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:36.685 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:36 smithi026 ceph-mon[29210]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:37.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 -- 172.21.15.31:0/4202738127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 msgr2=0x7f02bc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 --2- 172.21.15.31:0/4202738127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc100b40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009910 tx=0x7f02ac009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 -- 172.21.15.31:0/4202738127 shutdown_connections 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 --2- 172.21.15.31:0/4202738127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc100b40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 -- 172.21.15.31:0/4202738127 >> 172.21.15.31:0/4202738127 conn(0x7f02bc0fbcc0 msgr2=0x7f02bc0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 -- 172.21.15.31:0/4202738127 shutdown_connections 2024-03-24T10:35:37.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.462+0000 7f02c0d90700 1 -- 172.21.15.31:0/4202738127 wait complete. 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02c0d90700 1 Processor -- start 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02c0d90700 1 -- start start 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02c0d90700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02c0d90700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc074f60 con 0x7f02bc100720 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02ba59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02ba59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:56886/0 (socket says 172.21.15.31:56886) 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02ba59c700 1 -- 172.21.15.31:0/2352305383 learned_addr learned my addr 172.21.15.31:0/2352305383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.463+0000 7f02ba59c700 1 -- 172.21.15.31:0/2352305383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0095f0 con 0x7f02bc100720 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02ba59c700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f02ac006010 tx=0x7f02ac0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ac01d070 con 0x7f02bc100720 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02ac00be20 con 0x7f02bc100720 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc0731a0 con 0x7f02bc100720 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ac022de0 con 0x7f02bc100720 2024-03-24T10:35:37.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc073640 con 0x7f02bc100720 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.464+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f02ac00b960 con 0x7f02bc100720 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.465+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc106e80 con 0x7f02bc100720 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.465+0000 7f02ab7fe700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 0x7f02a403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.465+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f02ac04c930 con 0x7f02bc100720 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.465+0000 7f02b9d9b700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 0x7f02a403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.465+0000 7f02b9d9b700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 0x7f02a403ace0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02b0006fd0 tx=0x7f02b0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.470+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f02ac0226c0 con 0x7f02bc100720 2024-03-24T10:35:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.686+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 --> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi026:172.21.15.26=smithi026;smithi031:172.21.15.31=smithi031", "target": ["mon-mgr", ""]}) v1 -- 0x7f02bc18f4a0 con 0x7f02a4038820 2024-03-24T10:35:37.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.691+0000 7f02ab7fe700 1 -- 172.21.15.31:0/2352305383 <== mgr.14164 v2:172.21.15.26:6800/1986399491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f02bc18f4a0 con 0x7f02a4038820 2024-03-24T10:35:37.692 INFO:teuthology.orchestra.run.smithi031.stdout:Scheduled mon update... 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 msgr2=0x7f02a403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 0x7f02a403ace0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f02b0006fd0 tx=0x7f02b0006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 msgr2=0x7f02bc074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f02ac006010 tx=0x7f02ac0050b0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 shutdown_connections 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f02a4038820 0x7f02a403ace0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 --2- 172.21.15.31:0/2352305383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02bc100720 0x7f02bc074a20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 >> 172.21.15.31:0/2352305383 conn(0x7f02bc0fbcc0 msgr2=0x7f02bc0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 shutdown_connections 2024-03-24T10:35:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:37.693+0000 7f02c0d90700 1 -- 172.21.15.31:0/2352305383 wait complete. 2024-03-24T10:35:38.251 DEBUG:teuthology.orchestra.run.smithi031:mon.smithi031> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi031.service 2024-03-24T10:35:38.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:38.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:38.483 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:38.597 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:38 smithi026 ceph-mon[29210]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:38.597 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:38 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:39.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:39 smithi026 ceph-mon[29210]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi026:172.21.15.26=smithi026;smithi031:172.21.15.31=smithi031", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:39.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:39 smithi026 ceph-mon[29210]: Saving service mon spec with placement smithi026:172.21.15.26=smithi026;smithi031:172.21.15.31=smithi031;count:2 2024-03-24T10:35:39.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 -- 172.21.15.31:0/4115151069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 msgr2=0x7f83780ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 --2- 172.21.15.31:0/4115151069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f83780ffa60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8368009910 tx=0x7f8368009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:39.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 -- 172.21.15.31:0/4115151069 shutdown_connections 2024-03-24T10:35:39.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 --2- 172.21.15.31:0/4115151069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f83780ffa60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 -- 172.21.15.31:0/4115151069 >> 172.21.15.31:0/4115151069 conn(0x7f83780fabe0 msgr2=0x7f83780fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 -- 172.21.15.31:0/4115151069 shutdown_connections 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.773+0000 7f837e7b0700 1 -- 172.21.15.31:0/4115151069 wait complete. 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f837e7b0700 1 Processor -- start 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f837e7b0700 1 -- start start 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f837e7b0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f837e7b0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8378196100 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f8377fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f8377fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:56902/0 (socket says 172.21.15.31:56902) 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.774+0000 7f8377fff700 1 -- 172.21.15.31:0/2852154282 learned_addr learned my addr 172.21.15.31:0/2852154282 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f8377fff700 1 -- 172.21.15.31:0/2852154282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83680095f0 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f8377fff700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8368006010 tx=0x7f8368009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f836801d070 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8378196300 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8368005dc0 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8368022e90 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.775+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83781967a0 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.776+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f836800b960 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.776+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837818f6f0 con 0x7f83780ff640 2024-03-24T10:35:39.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.776+0000 7f83757fa700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 0x7f836403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.776+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f836804c8c0 con 0x7f83780ff640 2024-03-24T10:35:39.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.776+0000 7f83777fe700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 0x7f836403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.777+0000 7f83777fe700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 0x7f836403ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8360006fd0 tx=0x7f8360006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:39.781+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8368022750 con 0x7f83780ff640 2024-03-24T10:35:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.068+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83780623e0 con 0x7f83780ff640 2024-03-24T10:35:40.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.068+0000 7f83757fa700 1 -- 172.21.15.31:0/2852154282 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8368027020 con 0x7f83780ff640 2024-03-24T10:35:40.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:40.069 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 msgr2=0x7f836403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 0x7f836403ac20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8360006fd0 tx=0x7f8360006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 msgr2=0x7f8378195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8368006010 tx=0x7f8368009f80 comp rx=0 tx=0).stop 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 shutdown_connections 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8364038760 0x7f836403ac20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 --2- 172.21.15.31:0/2852154282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83780ff640 0x7f8378195bc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 >> 172.21.15.31:0/2852154282 conn(0x7f83780fabe0 msgr2=0x7f8378068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 shutdown_connections 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:40.070+0000 7f837e7b0700 1 -- 172.21.15.31:0/2852154282 wait complete. 2024-03-24T10:35:40.073 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T10:35:40.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:40.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:40 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2852154282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:41.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:41.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:41.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:41 smithi026 ceph-mon[29210]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T10:35:41.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:41 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T10:35:41.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:41 smithi026 ceph-mon[29210]: Deploying daemon grafana.smithi026 on smithi026 2024-03-24T10:35:41.829 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:42.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:42 smithi026 ceph-mon[29210]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- 172.21.15.31:0/3927645974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 msgr2=0x7f9d4c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 --2- 172.21.15.31:0/3927645974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c100b40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00b0a0 tx=0x7f9d3c00b3b0 comp rx=0 tx=0).stop 2024-03-24T10:35:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- 172.21.15.31:0/3927645974 shutdown_connections 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 --2- 172.21.15.31:0/3927645974 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c100b40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- 172.21.15.31:0/3927645974 >> 172.21.15.31:0/3927645974 conn(0x7f9d4c0fbcc0 msgr2=0x7f9d4c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- 172.21.15.31:0/3927645974 shutdown_connections 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- 172.21.15.31:0/3927645974 wait complete. 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 Processor -- start 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.441+0000 7f9d536b0700 1 -- start start 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d536b0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d536b0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c196100 con 0x7f9d4c100720 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d5144c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d5144c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:47882/0 (socket says 172.21.15.31:47882) 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d5144c700 1 -- 172.21.15.31:0/2927144428 learned_addr learned my addr 172.21.15.31:0/2927144428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d5144c700 1 -- 172.21.15.31:0/2927144428 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009d00 con 0x7f9d4c100720 2024-03-24T10:35:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d5144c700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0062a0 tx=0x7f9d3c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d3c00e040 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d3c003d70 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c196300 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.442+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d3c01c890 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.443+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c1967a0 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.443+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f9d3c012640 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.443+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c18f550 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.444+0000 7f9d4a7fc700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 0x7f9d3403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.444+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d3c04c950 con 0x7f9d4c100720 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.444+0000 7f9d50c4b700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 0x7f9d3403acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.444+0000 7f9d50c4b700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 0x7f9d3403acd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d40006fd0 tx=0x7f9d40006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.449+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d3c01a590 con 0x7f9d4c100720 2024-03-24T10:35:43.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.735+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d4c0623e0 con 0x7f9d4c100720 2024-03-24T10:35:43.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.735+0000 7f9d4a7fc700 1 -- 172.21.15.31:0/2927144428 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9d3c01a3b0 con 0x7f9d4c100720 2024-03-24T10:35:43.736 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:43.736 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:43.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 msgr2=0x7f9d3403acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 0x7f9d3403acd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d40006fd0 tx=0x7f9d40006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 msgr2=0x7f9d4c195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0062a0 tx=0x7f9d3c00bd40 comp rx=0 tx=0).stop 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 shutdown_connections 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9d34038810 0x7f9d3403acd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 --2- 172.21.15.31:0/2927144428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d4c100720 0x7f9d4c195bc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 >> 172.21.15.31:0/2927144428 conn(0x7f9d4c0fbcc0 msgr2=0x7f9d4c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 shutdown_connections 2024-03-24T10:35:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:43.737+0000 7f9d536b0700 1 -- 172.21.15.31:0/2927144428 wait complete. 2024-03-24T10:35:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:44.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:44 smithi026 ceph-mon[29210]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:44.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:44 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:44.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:44 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2927144428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:45.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:45.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:45.525 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:46.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:46 smithi026 ceph-mon[29210]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 -- 172.21.15.31:0/2440429055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 msgr2=0x7f83f00ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 --2- 172.21.15.31:0/2440429055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f00ffb20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f83e000b0a0 tx=0x7f83e000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 -- 172.21.15.31:0/2440429055 shutdown_connections 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 --2- 172.21.15.31:0/2440429055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f00ffb20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 -- 172.21.15.31:0/2440429055 >> 172.21.15.31:0/2440429055 conn(0x7f83f00facc0 msgr2=0x7f83f00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 -- 172.21.15.31:0/2440429055 shutdown_connections 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.917+0000 7f83f7778700 1 -- 172.21.15.31:0/2440429055 wait complete. 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f7778700 1 Processor -- start 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f7778700 1 -- start start 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f7778700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f7778700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f01961b0 con 0x7f83f00ff700 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f5514700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f5514700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:47900/0 (socket says 172.21.15.31:47900) 2024-03-24T10:35:46.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f5514700 1 -- 172.21.15.31:0/2427900989 learned_addr learned my addr 172.21.15.31:0/2427900989 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:46.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.918+0000 7f83f5514700 1 -- 172.21.15.31:0/2427900989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0009d00 con 0x7f83f00ff700 2024-03-24T10:35:46.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83f5514700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f83e00062a0 tx=0x7f83e000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e000e040 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f01963b0 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83e0003d70 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e001c890 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.919+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f0196850 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f018f7a0 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f83e001cb50 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83ee7fc700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 0x7f83d803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f83e004ca00 con 0x7f83f00ff700 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83f4d13700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 0x7f83d803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.920+0000 7f83f4d13700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 0x7f83d803ac80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006fd0 tx=0x7f83e4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:46.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:46.928+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f83e001a590 con 0x7f83f00ff700 2024-03-24T10:35:47.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.215+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83f00623e0 con 0x7f83f00ff700 2024-03-24T10:35:47.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.215+0000 7f83ee7fc700 1 -- 172.21.15.31:0/2427900989 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f83e001a3b0 con 0x7f83f00ff700 2024-03-24T10:35:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:47.216 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 msgr2=0x7f83d803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 0x7f83d803ac80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006fd0 tx=0x7f83e4006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 msgr2=0x7f83f0195c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f83e00062a0 tx=0x7f83e000bd40 comp rx=0 tx=0).stop 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 shutdown_connections 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f83d80387c0 0x7f83d803ac80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 --2- 172.21.15.31:0/2427900989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83f00ff700 0x7f83f0195c70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 >> 172.21.15.31:0/2427900989 conn(0x7f83f00facc0 msgr2=0x7f83f0068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 shutdown_connections 2024-03-24T10:35:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:47.217+0000 7f83f7778700 1 -- 172.21.15.31:0/2427900989 wait complete. 2024-03-24T10:35:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:47.557 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:47 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2427900989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:48.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:48 smithi026 ceph-mon[29210]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:48.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:48.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:49.101 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 -- 172.21.15.31:0/3385564731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 msgr2=0x7f87241035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 --2- 172.21.15.31:0/3385564731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f87241035a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f871400b0a0 tx=0x7f871400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 -- 172.21.15.31:0/3385564731 shutdown_connections 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 --2- 172.21.15.31:0/3385564731 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f87241035a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 -- 172.21.15.31:0/3385564731 >> 172.21.15.31:0/3385564731 conn(0x7f87240fabc0 msgr2=0x7f87240fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 -- 172.21.15.31:0/3385564731 shutdown_connections 2024-03-24T10:35:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.589+0000 7f8728ffd700 1 -- 172.21.15.31:0/3385564731 wait complete. 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f8728ffd700 1 Processor -- start 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f8728ffd700 1 -- start start 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f8728ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f8728ffd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87241982c0 con 0x7f87241011b0 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f872259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f872259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:57490/0 (socket says 172.21.15.31:57490) 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.590+0000 7f872259c700 1 -- 172.21.15.31:0/145465074 learned_addr learned my addr 172.21.15.31:0/145465074 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f872259c700 1 -- 172.21.15.31:0/145465074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8714009d00 con 0x7f87241011b0 2024-03-24T10:35:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f872259c700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87140062a0 tx=0x7f871400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f871400e040 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87241984c0 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8714003d70 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f871401c8d0 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.591+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8724198960 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.592+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f871401cb90 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.592+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8724191950 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.592+0000 7f871b7fe700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 0x7f871003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.592+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f871404ca40 con 0x7f87241011b0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.593+0000 7f8721d9b700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 0x7f871003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.593+0000 7f8721d9b700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 0x7f871003acd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f870c006fd0 tx=0x7f870c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:50.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.601+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f871401a590 con 0x7f87241011b0 2024-03-24T10:35:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:50 smithi026 ceph-mon[29210]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.888+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87240623e0 con 0x7f87241011b0 2024-03-24T10:35:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.888+0000 7f871b7fe700 1 -- 172.21.15.31:0/145465074 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f871401a3b0 con 0x7f87241011b0 2024-03-24T10:35:50.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:50.889 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 msgr2=0x7f871003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 0x7f871003acd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f870c006fd0 tx=0x7f870c006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 msgr2=0x7f8724197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87140062a0 tx=0x7f871400bd40 comp rx=0 tx=0).stop 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 shutdown_connections 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f8710038810 0x7f871003acd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 --2- 172.21.15.31:0/145465074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87241011b0 0x7f8724197d80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 >> 172.21.15.31:0/145465074 conn(0x7f87240fabc0 msgr2=0x7f87240fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.890+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 shutdown_connections 2024-03-24T10:35:50.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:50.891+0000 7f8728ffd700 1 -- 172.21.15.31:0/145465074 wait complete. 2024-03-24T10:35:50.892 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:51.629 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:51 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/145465074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:52.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:52.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:52.593 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:52 smithi026 ceph-mon[29210]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:52.652 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:52.697 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:54.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 -- 172.21.15.31:0/1403084882 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 msgr2=0x7f88380ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 --2- 172.21.15.31:0/1403084882 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380ffa60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f882800b0a0 tx=0x7f882800b3b0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 -- 172.21.15.31:0/1403084882 shutdown_connections 2024-03-24T10:35:54.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 --2- 172.21.15.31:0/1403084882 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380ffa60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 -- 172.21.15.31:0/1403084882 >> 172.21.15.31:0/1403084882 conn(0x7f88380fac00 msgr2=0x7f88380fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 -- 172.21.15.31:0/1403084882 shutdown_connections 2024-03-24T10:35:54.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.137+0000 7f883d1d4700 1 -- 172.21.15.31:0/1403084882 wait complete. 2024-03-24T10:35:54.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f883d1d4700 1 Processor -- start 2024-03-24T10:35:54.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f883d1d4700 1 -- start start 2024-03-24T10:35:54.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f883d1d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f883d1d4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380fcf60 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f8836d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f8836d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:57512/0 (socket says 172.21.15.31:57512) 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f8836d9d700 1 -- 172.21.15.31:0/4196240269 learned_addr learned my addr 172.21.15.31:0/4196240269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.138+0000 7f8836d9d700 1 -- 172.21.15.31:0/4196240269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8828009d00 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f8836d9d700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f88280062a0 tx=0x7f882800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882800e040 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88380fb1a0 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8828003d70 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882801c890 con 0x7f88380ff640 2024-03-24T10:35:54.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.139+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88380fb640 con 0x7f88380ff640 2024-03-24T10:35:54.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f882801cb50 con 0x7f88380ff640 2024-03-24T10:35:54.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f882ffff700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 0x7f882403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f882804ca90 con 0x7f88380ff640 2024-03-24T10:35:54.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8838105da0 con 0x7f88380ff640 2024-03-24T10:35:54.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f883659c700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 0x7f882403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.140+0000 7f883659c700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 0x7f882403ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.145+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8828003f90 con 0x7f88380ff640 2024-03-24T10:35:54.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.431+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88380623e0 con 0x7f88380ff640 2024-03-24T10:35:54.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.432+0000 7f882ffff700 1 -- 172.21.15.31:0/4196240269 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f88380623e0 con 0x7f88380ff640 2024-03-24T10:35:54.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:54.433 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 msgr2=0x7f882403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 0x7f882403ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 msgr2=0x7f88380fca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f88280062a0 tx=0x7f882800bd40 comp rx=0 tx=0).stop 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 shutdown_connections 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f88240387c0 0x7f882403ac80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 --2- 172.21.15.31:0/4196240269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88380ff640 0x7f88380fca20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 >> 172.21.15.31:0/4196240269 conn(0x7f88380fac00 msgr2=0x7f8838068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 shutdown_connections 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:54.435+0000 7f883d1d4700 1 -- 172.21.15.31:0/4196240269 wait complete. 2024-03-24T10:35:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:54.611 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:54 smithi026 ceph-mon[29210]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/4196240269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:55.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:55 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:56.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:56.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:56.218 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:56.264 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:56.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:56 smithi026 ceph-mon[29210]: Deploying daemon prometheus.smithi026 on smithi026 2024-03-24T10:35:56.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:56 smithi026 ceph-mon[29210]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:57.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 -- 172.21.15.31:0/2880069478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 msgr2=0x7f10400ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 --2- 172.21.15.31:0/2880069478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f10400ffa60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1030009910 tx=0x7f1030009c20 comp rx=0 tx=0).stop 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 -- 172.21.15.31:0/2880069478 shutdown_connections 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 --2- 172.21.15.31:0/2880069478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f10400ffa60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 -- 172.21.15.31:0/2880069478 >> 172.21.15.31:0/2880069478 conn(0x7f10400fabe0 msgr2=0x7f10400fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 -- 172.21.15.31:0/2880069478 shutdown_connections 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 -- 172.21.15.31:0/2880069478 wait complete. 2024-03-24T10:35:57.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.664+0000 7f104573c700 1 Processor -- start 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f104573c700 1 -- start start 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f104573c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f104573c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1040198330 con 0x7f10400ff640 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f103effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f103effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:57526/0 (socket says 172.21.15.31:57526) 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f103effd700 1 -- 172.21.15.31:0/1816448770 learned_addr learned my addr 172.21.15.31:0/1816448770 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f103effd700 1 -- 172.21.15.31:0/1816448770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300095f0 con 0x7f10400ff640 2024-03-24T10:35:57.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.665+0000 7f103effd700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1030006010 tx=0x7f10300050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103001d070 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f103000be20 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1040198530 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1030022de0 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10401989d0 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f103000b960 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.666+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1040191f90 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.667+0000 7f1027fff700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 0x7f102803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.667+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f103004c980 con 0x7f10400ff640 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.667+0000 7f103e7fc700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 0x7f102803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.667+0000 7f103e7fc700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 0x7f102803ac70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.672+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f10300226c0 con 0x7f10400ff640 2024-03-24T10:35:57.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.960+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10400623e0 con 0x7f10400ff640 2024-03-24T10:35:57.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.961+0000 7f1027fff700 1 -- 172.21.15.31:0/1816448770 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1030027030 con 0x7f10400ff640 2024-03-24T10:35:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:35:57.961 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 msgr2=0x7f102803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 0x7f102803ac70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034006e40 comp rx=0 tx=0).stop 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 msgr2=0x7f1040197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1030006010 tx=0x7f10300050b0 comp rx=0 tx=0).stop 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 shutdown_connections 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f10280387b0 0x7f102803ac70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 --2- 172.21.15.31:0/1816448770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10400ff640 0x7f1040197df0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:57.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 >> 172.21.15.31:0/1816448770 conn(0x7f10400fabe0 msgr2=0x7f104018f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:57.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 shutdown_connections 2024-03-24T10:35:57.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:35:57.963+0000 7f104573c700 1 -- 172.21.15.31:0/1816448770 wait complete. 2024-03-24T10:35:57.964 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:35:58.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:58 smithi026 ceph-mon[29210]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:35:58.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:58 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/1816448770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:35:59.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:35:59.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:35:59.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:35:59 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:35:59.791 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:35:59.837 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:00.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:00 smithi026 ceph-mon[29210]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 -- 172.21.15.31:0/3452290427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 msgr2=0x7f37d80fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 --2- 172.21.15.31:0/3452290427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d80fee30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009910 tx=0x7f37c8009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 -- 172.21.15.31:0/3452290427 shutdown_connections 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 --2- 172.21.15.31:0/3452290427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d80fee30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 -- 172.21.15.31:0/3452290427 >> 172.21.15.31:0/3452290427 conn(0x7f37d80fa370 msgr2=0x7f37d80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 -- 172.21.15.31:0/3452290427 shutdown_connections 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.209+0000 7f37dde5a700 1 -- 172.21.15.31:0/3452290427 wait complete. 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37dde5a700 1 Processor -- start 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37dde5a700 1 -- start start 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37dde5a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37dde5a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d81960e0 con 0x7f37d80fea10 2024-03-24T10:36:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:49888/0 (socket says 172.21.15.31:49888) 2024-03-24T10:36:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.210+0000 7f37d77fe700 1 -- 172.21.15.31:0/294952311 learned_addr learned my addr 172.21.15.31:0/294952311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37d77fe700 1 -- 172.21.15.31:0/294952311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80095f0 con 0x7f37d80fea10 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37d77fe700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006010 tx=0x7f37c8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c801d070 con 0x7f37d80fea10 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37c8005dc0 con 0x7f37d80fea10 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d8196340 con 0x7f37d80fea10 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c8022e90 con 0x7f37d80fea10 2024-03-24T10:36:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.211+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d8199010 con 0x7f37d80fea10 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.212+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f37c800b960 con 0x7f37d80fea10 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.212+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d818f500 con 0x7f37d80fea10 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.212+0000 7f37d4ff9700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 0x7f37c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.212+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f37c804c840 con 0x7f37d80fea10 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.212+0000 7f37d6ffd700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 0x7f37c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.213+0000 7f37d6ffd700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 0x7f37c003ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37cc006fd0 tx=0x7f37cc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.218+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f37c800bd00 con 0x7f37d80fea10 2024-03-24T10:36:01.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.504+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37d80623e0 con 0x7f37d80fea10 2024-03-24T10:36:01.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.505+0000 7f37d4ff9700 1 -- 172.21.15.31:0/294952311 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f37c8027020 con 0x7f37d80fea10 2024-03-24T10:36:01.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:01.505 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:01.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 msgr2=0x7f37c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 0x7f37c003ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37cc006fd0 tx=0x7f37cc006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:01.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 msgr2=0x7f37d8195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006010 tx=0x7f37c8009f80 comp rx=0 tx=0).stop 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 shutdown_connections 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f37c0038760 0x7f37c003ac20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 --2- 172.21.15.31:0/294952311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37d80fea10 0x7f37d8195ba0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 >> 172.21.15.31:0/294952311 conn(0x7f37d80fa370 msgr2=0x7f37d80fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 shutdown_connections 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:01.507+0000 7f37dde5a700 1 -- 172.21.15.31:0/294952311 wait complete. 2024-03-24T10:36:01.508 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:02.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:02 smithi026 ceph-mon[29210]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:02.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:02 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/294952311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:03.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:03.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:03.302 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:03.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:04.629 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:04 smithi026 ceph-mon[29210]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 -- 172.21.15.31:0/2911922931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 msgr2=0x7f9a7c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2911922931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c100b40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009910 tx=0x7f9a74009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 -- 172.21.15.31:0/2911922931 shutdown_connections 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2911922931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c100b40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 -- 172.21.15.31:0/2911922931 >> 172.21.15.31:0/2911922931 conn(0x7f9a7c0fbcc0 msgr2=0x7f9a7c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 -- 172.21.15.31:0/2911922931 shutdown_connections 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.788+0000 7f9a839d6700 1 -- 172.21.15.31:0/2911922931 wait complete. 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a839d6700 1 Processor -- start 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a839d6700 1 -- start start 2024-03-24T10:36:04.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a839d6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a839d6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c074f60 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a81772700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a81772700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:49914/0 (socket says 172.21.15.31:49914) 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a81772700 1 -- 172.21.15.31:0/2389740861 learned_addr learned my addr 172.21.15.31:0/2389740861 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a81772700 1 -- 172.21.15.31:0/2389740861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740095f0 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a81772700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a74006010 tx=0x7f9a74009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7401d070 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a74005dc0 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c0731a0 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.789+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a74022e90 con 0x7f9a7c100720 2024-03-24T10:36:04.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.790+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c073640 con 0x7f9a7c100720 2024-03-24T10:36:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.790+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f9a7400b960 con 0x7f9a7c100720 2024-03-24T10:36:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.790+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c106e80 con 0x7f9a7c100720 2024-03-24T10:36:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.791+0000 7f9a7a7fc700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 0x7f9a6403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.791+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9a7404c800 con 0x7f9a7c100720 2024-03-24T10:36:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.791+0000 7f9a80f71700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 0x7f9a6403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.791+0000 7f9a80f71700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 0x7f9a6403ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:04.796+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9a74022750 con 0x7f9a7c100720 2024-03-24T10:36:05.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.078+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a7c0623e0 con 0x7f9a7c100720 2024-03-24T10:36:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.079+0000 7f9a7a7fc700 1 -- 172.21.15.31:0/2389740861 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9a74027020 con 0x7f9a7c100720 2024-03-24T10:36:05.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:05.079 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 msgr2=0x7f9a6403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 0x7f9a6403ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 msgr2=0x7f9a7c074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a74006010 tx=0x7f9a74009f80 comp rx=0 tx=0).stop 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 shutdown_connections 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f9a64038760 0x7f9a6403ac20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 --2- 172.21.15.31:0/2389740861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a7c100720 0x7f9a7c074a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 >> 172.21.15.31:0/2389740861 conn(0x7f9a7c0fbcc0 msgr2=0x7f9a7c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 shutdown_connections 2024-03-24T10:36:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:05.081+0000 7f9a839d6700 1 -- 172.21.15.31:0/2389740861 wait complete. 2024-03-24T10:36:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:05.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:05 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2389740861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:06 smithi026 ceph-mon[29210]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:06 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T10:36:06.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:06.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:06.908 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:07.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:07 smithi026 ceph-mon[29210]: from='mgr.14164 172.21.15.26:0/981395991' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T10:36:07.947 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:07 smithi026 ceph-mon[29210]: mgrmap e14: smithi026.bbjcru(active, since 63s) 2024-03-24T10:36:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 -- 172.21.15.31:0/2367961992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 msgr2=0x7f48900ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2367961992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f48900ffa50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f488000b0a0 tx=0x7f488000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 -- 172.21.15.31:0/2367961992 shutdown_connections 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2367961992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f48900ffa50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 -- 172.21.15.31:0/2367961992 >> 172.21.15.31:0/2367961992 conn(0x7f48900fabd0 msgr2=0x7f48900fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 -- 172.21.15.31:0/2367961992 shutdown_connections 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.413+0000 7f4897f7b700 1 -- 172.21.15.31:0/2367961992 wait complete. 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4897f7b700 1 Processor -- start 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4897f7b700 1 -- start start 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4897f7b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4897f7b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890069a70 con 0x7f48900ff630 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4895d17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4895d17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:49936/0 (socket says 172.21.15.31:49936) 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4895d17700 1 -- 172.21.15.31:0/2861045620 learned_addr learned my addr 172.21.15.31:0/2861045620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4895d17700 1 -- 172.21.15.31:0/2861045620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4880009d00 con 0x7f48900ff630 2024-03-24T10:36:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.414+0000 7f4895d17700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f488000bc90 tx=0x7f488000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488000e040 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4880003d50 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890198890 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488001c8f0 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4890198d30 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 14) v1 ==== 45618+0+0 (secure 0 0 0) 0x7f488001cbb0 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.415+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489018e460 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.416+0000 7f488effd700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 0x7f487803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.416+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f488004ca60 con 0x7f48900ff630 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.416+0000 7f4895516700 1 -- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 msgr2=0x7f487803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/1986399491 2024-03-24T10:36:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.416+0000 7f4895516700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 0x7f487803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:36:08.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.421+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f488001a590 con 0x7f48900ff630 2024-03-24T10:36:08.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.617+0000 7f4895516700 1 -- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 msgr2=0x7f487803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/1986399491 2024-03-24T10:36:08.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.617+0000 7f4895516700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 0x7f487803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:36:08.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.709+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48900623e0 con 0x7f48900ff630 2024-03-24T10:36:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.710+0000 7f488effd700 1 -- 172.21.15.31:0/2861045620 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f488001a3b0 con 0x7f48900ff630 2024-03-24T10:36:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:08.710 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:08.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 msgr2=0x7f487803ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 0x7f487803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 msgr2=0x7f4890069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f488000bc90 tx=0x7f488000bd70 comp rx=0 tx=0).stop 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 shutdown_connections 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f4878038860 0x7f487803ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 --2- 172.21.15.31:0/2861045620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48900ff630 0x7f4890069530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 >> 172.21.15.31:0/2861045620 conn(0x7f48900fabd0 msgr2=0x7f48900fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 shutdown_connections 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:08.712+0000 7f4897f7b700 1 -- 172.21.15.31:0/2861045620 wait complete. 2024-03-24T10:36:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:08.932 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:08 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2861045620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:10.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:10.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:10.428 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2931078225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 msgr2=0x7f1c94103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2931078225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94103660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009910 tx=0x7f1c88009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2931078225 shutdown_connections 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2931078225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94103660 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2931078225 >> 172.21.15.31:0/2931078225 conn(0x7f1c940fabe0 msgr2=0x7f1c940fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2931078225 shutdown_connections 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.958+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2931078225 wait complete. 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c9af1d700 1 Processor -- start 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c9af1d700 1 -- start start 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c9af1d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c9af1d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c94100fe0 con 0x7f1c94101270 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c98cb9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c98cb9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:58996/0 (socket says 172.21.15.31:58996) 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c98cb9700 1 -- 172.21.15.31:0/2723680473 learned_addr learned my addr 172.21.15.31:0/2723680473 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c98cb9700 1 -- 172.21.15.31:0/2723680473 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880095f0 con 0x7f1c94101270 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.959+0000 7f1c98cb9700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c88006010 tx=0x7f1c880050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.960+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c8801d070 con 0x7f1c94101270 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.960+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c8800be20 con 0x7f1c94101270 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.960+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c940ff220 con 0x7f1c94101270 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.960+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c8800f460 con 0x7f1c94101270 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.960+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c940ff6c0 con 0x7f1c94101270 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 14) v1 ==== 45618+0+0 (secure 0 0 0) 0x7f1c8800f720 con 0x7f1c94101270 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c91ffb700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 0x7f1c8403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c8804cc70 con 0x7f1c94101270 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c93fff700 1 -- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 msgr2=0x7f1c8403ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/1986399491 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c94106ea0 con 0x7f1c94101270 2024-03-24T10:36:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.961+0000 7f1c93fff700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 0x7f1c8403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:36:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:11.966+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1c88027070 con 0x7f1c94101270 2024-03-24T10:36:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.162+0000 7f1c93fff700 1 -- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 msgr2=0x7f1c8403ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/1986399491 2024-03-24T10:36:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.162+0000 7f1c93fff700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 0x7f1c8403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:36:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.248+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c940623e0 con 0x7f1c94101270 2024-03-24T10:36:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.248+0000 7f1c91ffb700 1 -- 172.21.15.31:0/2723680473 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1c940623e0 con 0x7f1c94101270 2024-03-24T10:36:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:12.249 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.250+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 msgr2=0x7f1c8403ace0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:36:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.250+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 0x7f1c8403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.250+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 msgr2=0x7f1c94100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.250+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c88006010 tx=0x7f1c880050b0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 shutdown_connections 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:6800/1986399491,v1:172.21.15.26:6801/1986399491] conn(0x7f1c84038820 0x7f1c8403ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 --2- 172.21.15.31:0/2723680473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c94101270 0x7f1c94100aa0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 >> 172.21.15.31:0/2723680473 conn(0x7f1c940fabe0 msgr2=0x7f1c940fb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 shutdown_connections 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:12.251+0000 7f1c9af1d700 1 -- 172.21.15.31:0/2723680473 wait complete. 2024-03-24T10:36:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:12.696 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:12 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2723680473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:13.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:13.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: Active manager daemon smithi026.bbjcru restarted 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: Activating manager daemon smithi026.bbjcru 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: mgrmap e15: smithi026.bbjcru(active, starting, since 0.00304002s) 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi026.bbjcru", "id": "smithi026.bbjcru"}]: dispatch 2024-03-24T10:36:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:36:13.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:36:13.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:36:13.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: Manager daemon smithi026.bbjcru is now available 2024-03-24T10:36:13.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:13.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:14.045 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi026.bbjcru/trash_purge_schedule"}]: dispatch 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:14.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:14 smithi026 ceph-mon[29210]: mgrmap e16: smithi026.bbjcru(active, since 1.00558s) 2024-03-24T10:36:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 -- 172.21.15.31:0/2135381468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 msgr2=0x7f71fc103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 --2- 172.21.15.31:0/2135381468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc103640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009910 tx=0x7f71f0009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 -- 172.21.15.31:0/2135381468 shutdown_connections 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 --2- 172.21.15.31:0/2135381468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc103640 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 -- 172.21.15.31:0/2135381468 >> 172.21.15.31:0/2135381468 conn(0x7f71fc0fabe0 msgr2=0x7f71fc0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 -- 172.21.15.31:0/2135381468 shutdown_connections 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.635+0000 7f7200a6b700 1 -- 172.21.15.31:0/2135381468 wait complete. 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.636+0000 7f7200a6b700 1 Processor -- start 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.636+0000 7f7200a6b700 1 -- start start 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.636+0000 7f7200a6b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.636+0000 7f7200a6b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc100fc0 con 0x7f71fc101250 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71fa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71fa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:59006/0 (socket says 172.21.15.31:59006) 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71fa59c700 1 -- 172.21.15.31:0/3306044112 learned_addr learned my addr 172.21.15.31:0/3306044112 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71fa59c700 1 -- 172.21.15.31:0/3306044112 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f00095f0 con 0x7f71fc101250 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71fa59c700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009ed0 tx=0x7f71f0004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71f001d070 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71f000be20 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71fc0ff200 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71f000f460 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.637+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc0ff6a0 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.638+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) v1 ==== 45469+0+0 (secure 0 0 0) 0x7f71f000f650 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.638+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71fc106ea0 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.638+0000 7f71eb7fe700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 0x7f71e403ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.638+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f71f004c8c0 con 0x7f71fc101250 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.638+0000 7f71f9d9b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 0x7f71e403ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.639+0000 7f71f9d9b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 0x7f71e403ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.643+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f71f002a360 con 0x7f71fc101250 2024-03-24T10:36:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.929+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71fc062430 con 0x7f71fc101250 2024-03-24T10:36:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.929+0000 7f71eb7fe700 1 -- 172.21.15.31:0/3306044112 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f71f0027030 con 0x7f71fc101250 2024-03-24T10:36:15.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:15.930 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 msgr2=0x7f71e403ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 0x7f71e403ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 msgr2=0x7f71fc100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009ed0 tx=0x7f71f0004690 comp rx=0 tx=0).stop 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 shutdown_connections 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f71e4038660 0x7f71e403ab20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 --2- 172.21.15.31:0/3306044112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71fc101250 0x7f71fc100a80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.931+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 >> 172.21.15.31:0/3306044112 conn(0x7f71fc0fabe0 msgr2=0x7f71fc0fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.932+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 shutdown_connections 2024-03-24T10:36:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:15.932+0000 7f7200a6b700 1 -- 172.21.15.31:0/3306044112 wait complete. 2024-03-24T10:36:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:16.101 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:16.102 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: [24/Mar/2024:10:36:15] ENGINE Bus STARTING 2024-03-24T10:36:16.102 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: [24/Mar/2024:10:36:15] ENGINE Serving on https://172.21.15.26:7150 2024-03-24T10:36:16.102 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: [24/Mar/2024:10:36:15] ENGINE Serving on http://172.21.15.26:8765 2024-03-24T10:36:16.102 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: [24/Mar/2024:10:36:15] ENGINE Bus STARTED 2024-03-24T10:36:16.102 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:16.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:16 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/3306044112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:16.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:16 smithi026 ceph-mon[29210]: mgrmap e17: smithi026.bbjcru(active, since 2s) 2024-03-24T10:36:17.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:17.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:17.976 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:18.033 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T10:36:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:20.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.436+0000 7f82020ba700 1 -- 172.21.15.31:0/1476870458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 msgr2=0x7f81fc105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.436+0000 7f82020ba700 1 --2- 172.21.15.31:0/1476870458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc105a40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f81ec009910 tx=0x7f81ec009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.436+0000 7f82020ba700 1 -- 172.21.15.31:0/1476870458 shutdown_connections 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.436+0000 7f82020ba700 1 --2- 172.21.15.31:0/1476870458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc105a40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.436+0000 7f82020ba700 1 -- 172.21.15.31:0/1476870458 >> 172.21.15.31:0/1476870458 conn(0x7f81fc0fb0e0 msgr2=0x7f81fc0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 -- 172.21.15.31:0/1476870458 shutdown_connections 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 -- 172.21.15.31:0/1476870458 wait complete. 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 Processor -- start 2024-03-24T10:36:20.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 -- start start 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.437+0000 7f82020ba700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc0fd450 con 0x7f81fc105660 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81fb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81fb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:59024/0 (socket says 172.21.15.31:59024) 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81fb7fe700 1 -- 172.21.15.31:0/2030999302 learned_addr learned my addr 172.21.15.31:0/2030999302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81fb7fe700 1 -- 172.21.15.31:0/2030999302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0095f0 con 0x7f81fc105660 2024-03-24T10:36:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81fb7fe700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81ec005490 tx=0x7f81ec005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81ec01d070 con 0x7f81fc105660 2024-03-24T10:36:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81ec005dc0 con 0x7f81fc105660 2024-03-24T10:36:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc0fbad0 con 0x7f81fc105660 2024-03-24T10:36:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81ec022d60 con 0x7f81fc105660 2024-03-24T10:36:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.438+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc0fbf50 con 0x7f81fc105660 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.439+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f81ec00f460 con 0x7f81fc105660 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.439+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc101360 con 0x7f81fc105660 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.440+0000 7f81f8ff9700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 0x7f81e403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.440+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f81ec04c9f0 con 0x7f81fc105660 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.440+0000 7f81faffd700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 0x7f81e403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.440+0000 7f81faffd700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 0x7f81e403ad70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.445+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f81ec02a950 con 0x7f81fc105660 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: Updating smithi031:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: Updating smithi026:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi031", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi031", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-24T10:36:20.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:20.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.740+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81fc100d10 con 0x7f81fc105660 2024-03-24T10:36:20.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.741+0000 7f81f8ff9700 1 -- 172.21.15.31:0/2030999302 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f81ec027030 con 0x7f81fc105660 2024-03-24T10:36:20.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 msgr2=0x7f81e403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 0x7f81e403ad70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 msgr2=0x7f81fc0fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81ec005490 tx=0x7f81ec005570 comp rx=0 tx=0).stop 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 shutdown_connections 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f81e40388b0 0x7f81e403ad70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 --2- 172.21.15.31:0/2030999302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f81fc105660 0x7f81fc0fb590 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.743+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 >> 172.21.15.31:0/2030999302 conn(0x7f81fc0fb0e0 msgr2=0x7f81fc068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.744+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 shutdown_connections 2024-03-24T10:36:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:20.744+0000 7f82020ba700 1 -- 172.21.15.31:0/2030999302 wait complete. 2024-03-24T10:36:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:36:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:36:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: Updating smithi031:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:36:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: Updating smithi026:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:36:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: Deploying daemon ceph-exporter.smithi031 on smithi031 2024-03-24T10:36:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:21 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2030999302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:22.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:22.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:23.194 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 -- 172.21.15.31:0/343162971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 msgr2=0x7f7b38105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 --2- 172.21.15.31:0/343162971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38105a10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009910 tx=0x7f7b2c009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 -- 172.21.15.31:0/343162971 shutdown_connections 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 --2- 172.21.15.31:0/343162971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38105a10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 -- 172.21.15.31:0/343162971 >> 172.21.15.31:0/343162971 conn(0x7f7b38075110 msgr2=0x7f7b38075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 -- 172.21.15.31:0/343162971 shutdown_connections 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 -- 172.21.15.31:0/343162971 wait complete. 2024-03-24T10:36:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.996+0000 7f7b401f5700 1 Processor -- start 2024-03-24T10:36:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b401f5700 1 -- start start 2024-03-24T10:36:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b401f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b401f5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38068ce0 con 0x7f7b38105630 2024-03-24T10:36:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b3df91700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b3df91700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:45220/0 (socket says 172.21.15.31:45220) 2024-03-24T10:36:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b3df91700 1 -- 172.21.15.31:0/955610170 learned_addr learned my addr 172.21.15.31:0/955610170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.997+0000 7f7b3df91700 1 -- 172.21.15.31:0/955610170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0095f0 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b3df91700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c005490 tx=0x7f7b2c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b2c01d070 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b2c005700 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b38195170 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b2c003950 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.998+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b38195610 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.999+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f7b2c003c10 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.999+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b380fedf0 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.999+0000 7f7b2affd700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 0x7f7b2403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.999+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f7b2c04c560 con 0x7f7b38105630 2024-03-24T10:36:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:24.999+0000 7f7b3d790700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 0x7f7b2403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.000+0000 7f7b3d790700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 0x7f7b2403ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b34006fd0 tx=0x7f7b34006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.004+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b2c029950 con 0x7f7b38105630 2024-03-24T10:36:25.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.297+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b38100d00 con 0x7f7b38105630 2024-03-24T10:36:25.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.298+0000 7f7b2affd700 1 -- 172.21.15.31:0/955610170 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f7b2c026070 con 0x7f7b38105630 2024-03-24T10:36:25.298 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:25.298 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:25.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 msgr2=0x7f7b2403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 0x7f7b2403ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b34006fd0 tx=0x7f7b34006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 msgr2=0x7f7b38194c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c005490 tx=0x7f7b2c005570 comp rx=0 tx=0).stop 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 shutdown_connections 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7b240388c0 0x7f7b2403ad80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 --2- 172.21.15.31:0/955610170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b38105630 0x7f7b38194c30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 >> 172.21.15.31:0/955610170 conn(0x7f7b38075110 msgr2=0x7f7b380fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 shutdown_connections 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:25.300+0000 7f7b401f5700 1 -- 172.21.15.31:0/955610170 wait complete. 2024-03-24T10:36:25.301 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:26.114 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: Deploying daemon crash.smithi031 on smithi031 2024-03-24T10:36:26.115 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:25 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/955610170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:26.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:26.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:27.576 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:28 smithi026 ceph-mon[29210]: Deploying daemon node-exporter.smithi031 on smithi031 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/2169085375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 msgr2=0x7f7a28105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/2169085375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a28105a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009910 tx=0x7f7a18009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/2169085375 shutdown_connections 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/2169085375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a28105a30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/2169085375 >> 172.21.15.31:0/2169085375 conn(0x7f7a280fb0b0 msgr2=0x7f7a280fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/2169085375 shutdown_connections 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.024+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/2169085375 wait complete. 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a2d8b2700 1 Processor -- start 2024-03-24T10:36:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a2d8b2700 1 -- start start 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a2d8b2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a2d8b2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280fd420 con 0x7f7a28105650 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a26ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a26ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:45244/0 (socket says 172.21.15.31:45244) 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.025+0000 7f7a26ffd700 1 -- 172.21.15.31:0/191891092 learned_addr learned my addr 172.21.15.31:0/191891092 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a26ffd700 1 -- 172.21.15.31:0/191891092 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a180095f0 con 0x7f7a28105650 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a26ffd700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005950 tx=0x7f7a18005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1801d070 con 0x7f7a28105650 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a180056e0 con 0x7f7a28105650 2024-03-24T10:36:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a280fbaa0 con 0x7f7a28105650 2024-03-24T10:36:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a18022d90 con 0x7f7a28105650 2024-03-24T10:36:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.026+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a280fbf20 con 0x7f7a28105650 2024-03-24T10:36:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.027+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a28100d00 con 0x7f7a28105650 2024-03-24T10:36:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.027+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f7a18022600 con 0x7f7a28105650 2024-03-24T10:36:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.027+0000 7f7a2c8b0700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 0x7f7a1403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.027+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f7a1804c840 con 0x7f7a28105650 2024-03-24T10:36:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.027+0000 7f7a267fc700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 0x7f7a1403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.028+0000 7f7a267fc700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 0x7f7a1403ad70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a10006fd0 tx=0x7f7a10006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.032+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a1800f460 con 0x7f7a28105650 2024-03-24T10:36:29.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.326+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a280fc160 con 0x7f7a28105650 2024-03-24T10:36:29.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.326+0000 7f7a2c8b0700 1 -- 172.21.15.31:0/191891092 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f7a18027030 con 0x7f7a28105650 2024-03-24T10:36:29.327 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:29.327 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:29.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 msgr2=0x7f7a1403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 0x7f7a1403ad70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a10006fd0 tx=0x7f7a10006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:29.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 msgr2=0x7f7a280fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005950 tx=0x7f7a18005540 comp rx=0 tx=0).stop 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 shutdown_connections 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7a140388b0 0x7f7a1403ad70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 --2- 172.21.15.31:0/191891092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7a28105650 0x7f7a280fb560 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 >> 172.21.15.31:0/191891092 conn(0x7f7a280fb0b0 msgr2=0x7f7a28068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 shutdown_connections 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:29.329+0000 7f7a2d8b2700 1 -- 172.21.15.31:0/191891092 wait complete. 2024-03-24T10:36:29.330 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:36:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:29 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/191891092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:30.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:30.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:31.159 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:33.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 -- 172.21.15.31:0/3102717816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 msgr2=0x7f4ac0105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3102717816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0105a40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009910 tx=0x7f4ab0009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 -- 172.21.15.31:0/3102717816 shutdown_connections 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3102717816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0105a40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 -- 172.21.15.31:0/3102717816 >> 172.21.15.31:0/3102717816 conn(0x7f4ac00fb0e0 msgr2=0x7f4ac00fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 -- 172.21.15.31:0/3102717816 shutdown_connections 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.326+0000 7f4ac5667700 1 -- 172.21.15.31:0/3102717816 wait complete. 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4ac5667700 1 Processor -- start 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4ac5667700 1 -- start start 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4ac5667700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4ac5667700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0197210 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4abeffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4abeffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:46318/0 (socket says 172.21.15.31:46318) 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.327+0000 7f4abeffd700 1 -- 172.21.15.31:0/3757479124 learned_addr learned my addr 172.21.15.31:0/3757479124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4abeffd700 1 -- 172.21.15.31:0/3757479124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab00095f0 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4abeffd700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005490 tx=0x7f4ab0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ab001d070 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac0197470 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ab0005dc0 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ab0022d60 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.328+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac0197970 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.329+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f4ab000f460 con 0x7f4ac0105660 2024-03-24T10:36:33.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.329+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac00fcbf0 con 0x7f4ac0105660 2024-03-24T10:36:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.329+0000 7f4aa7fff700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 0x7f4aa803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.329+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4ab004cc90 con 0x7f4ac0105660 2024-03-24T10:36:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.329+0000 7f4abe7fc700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 0x7f4aa803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.330+0000 7f4abe7fc700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 0x7f4aa803ad70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4006fd0 tx=0x7f4ab4006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.335+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4ab002a950 con 0x7f4ac0105660 2024-03-24T10:36:33.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.619+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ac0068d80 con 0x7f4ac0105660 2024-03-24T10:36:33.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.619+0000 7f4aa7fff700 1 -- 172.21.15.31:0/3757479124 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4ab0027030 con 0x7f4ac0105660 2024-03-24T10:36:33.620 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:33.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.621+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 msgr2=0x7f4aa803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.621+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 0x7f4aa803ad70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4006fd0 tx=0x7f4ab4006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.621+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 msgr2=0x7f4ac0196cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.621+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005490 tx=0x7f4ab0005570 comp rx=0 tx=0).stop 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 shutdown_connections 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f4aa80388b0 0x7f4aa803ad70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 --2- 172.21.15.31:0/3757479124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac0105660 0x7f4ac0196cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 >> 172.21.15.31:0/3757479124 conn(0x7f4ac00fb0e0 msgr2=0x7f4ac00fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 shutdown_connections 2024-03-24T10:36:33.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:33.622+0000 7f4ac5667700 1 -- 172.21.15.31:0/3757479124 wait complete. 2024-03-24T10:36:33.623 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/3757479124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:34.954 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.ucwncv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.ucwncv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:36:34.955 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:35.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:35.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:35.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:35 smithi026 ceph-mon[29210]: Deploying daemon mgr.smithi031.ucwncv on smithi031 2024-03-24T10:36:35.968 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:36.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:36 smithi026 ceph-mon[29210]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.909+0000 7f043c9c5700 1 -- 172.21.15.31:0/3255876433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 msgr2=0x7f0438101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.909+0000 7f043c9c5700 1 --2- 172.21.15.31:0/3255876433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438101c70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0428009910 tx=0x7f0428009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.910+0000 7f043c9c5700 1 -- 172.21.15.31:0/3255876433 shutdown_connections 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.910+0000 7f043c9c5700 1 --2- 172.21.15.31:0/3255876433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438101c70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.910+0000 7f043c9c5700 1 -- 172.21.15.31:0/3255876433 >> 172.21.15.31:0/3255876433 conn(0x7f04380fb0e0 msgr2=0x7f04380fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.910+0000 7f043c9c5700 1 -- 172.21.15.31:0/3255876433 shutdown_connections 2024-03-24T10:36:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.910+0000 7f043c9c5700 1 -- 172.21.15.31:0/3255876433 wait complete. 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.911+0000 7f043c9c5700 1 Processor -- start 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.911+0000 7f043c9c5700 1 -- start start 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.911+0000 7f043c9c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.911+0000 7f043c9c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0438196540 con 0x7f0438101890 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f043659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f043659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:46354/0 (socket says 172.21.15.31:46354) 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f043659c700 1 -- 172.21.15.31:0/1534987538 learned_addr learned my addr 172.21.15.31:0/1534987538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f043659c700 1 -- 172.21.15.31:0/1534987538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280095f0 con 0x7f0438101890 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f043659c700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0428005490 tx=0x7f0428005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.912+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f042801d070 con 0x7f0438101890 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.913+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0428005700 con 0x7f0438101890 2024-03-24T10:36:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.913+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0438196740 con 0x7f0438101890 2024-03-24T10:36:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.913+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0428003950 con 0x7f0438101890 2024-03-24T10:36:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.913+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0438196aa0 con 0x7f0438101890 2024-03-24T10:36:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.914+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f0428003bc0 con 0x7f0438101890 2024-03-24T10:36:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.914+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04380693c0 con 0x7f0438101890 2024-03-24T10:36:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.914+0000 7f04277fe700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 0x7f042003ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.914+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f042804c630 con 0x7f0438101890 2024-03-24T10:36:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.914+0000 7f0435d9b700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 0x7f042003ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.915+0000 7f0435d9b700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 0x7f042003ae20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f042c006fd0 tx=0x7f042c006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:37.919+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0428029950 con 0x7f0438101890 2024-03-24T10:36:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.243+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f043819a8f0 con 0x7f0438101890 2024-03-24T10:36:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.244+0000 7f04277fe700 1 -- 172.21.15.31:0/1534987538 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0428026070 con 0x7f0438101890 2024-03-24T10:36:38.246 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:38.246 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 msgr2=0x7f042003ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 0x7f042003ae20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f042c006fd0 tx=0x7f042c006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 msgr2=0x7f0438196000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0428005490 tx=0x7f0428005570 comp rx=0 tx=0).stop 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 shutdown_connections 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0420038960 0x7f042003ae20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 --2- 172.21.15.31:0/1534987538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0438101890 0x7f0438196000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 >> 172.21.15.31:0/1534987538 conn(0x7f04380fb0e0 msgr2=0x7f0438102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 shutdown_connections 2024-03-24T10:36:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:38.246+0000 7f043c9c5700 1 -- 172.21.15.31:0/1534987538 wait complete. 2024-03-24T10:36:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:36:38.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:38.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: Deploying daemon mon.smithi031 on smithi031 2024-03-24T10:36:38.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:38.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:38 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/1534987538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:40.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:40.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:40.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:40.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:39 smithi026 ceph-mon[29210]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:36:42.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:41 smithi026 ceph-mon[29210]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 -- 172.21.15.31:0/2141300526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 msgr2=0x7fe2ec106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 --2- 172.21.15.31:0/2141300526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec106bd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009910 tx=0x7fe2e0009c20 comp rx=0 tx=0).stop 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 -- 172.21.15.31:0/2141300526 shutdown_connections 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 --2- 172.21.15.31:0/2141300526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec106bd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 -- 172.21.15.31:0/2141300526 >> 172.21.15.31:0/2141300526 conn(0x7fe2ec0fc300 msgr2=0x7fe2ec0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 -- 172.21.15.31:0/2141300526 shutdown_connections 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.303+0000 7fe2f404a700 1 -- 172.21.15.31:0/2141300526 wait complete. 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f404a700 1 Processor -- start 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f404a700 1 -- start start 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f404a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f404a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec1987b0 con 0x7fe2ec1067f0 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f1de6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f1de6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.31:46460/0 (socket says 172.21.15.31:46460) 2024-03-24T10:36:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.304+0000 7fe2f1de6700 1 -- 172.21.15.31:0/1235633905 learned_addr learned my addr 172.21.15.31:0/1235633905 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2f1de6700 1 -- 172.21.15.31:0/1235633905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00095f0 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2f1de6700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005490 tx=0x7fe2e0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2e001d070 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2e0005700 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec198a10 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2e00038e0 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.305+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec198e90 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.306+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7fe2e0003ba0 con 0x7fe2ec1067f0 2024-03-24T10:36:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.306+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec106170 con 0x7fe2ec1067f0 2024-03-24T10:36:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.306+0000 7fe2deffd700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 0x7fe2d803ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.306+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe2e004c6a0 con 0x7fe2ec1067f0 2024-03-24T10:36:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.306+0000 7fe2f15e5700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 0x7fe2d803ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.307+0000 7fe2f15e5700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 0x7fe2d803ad80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8006fd0 tx=0x7fe2e8006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:42.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.311+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe2e0029540 con 0x7fe2ec1067f0 2024-03-24T10:36:42.598 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 podman[37310]: 2024-03-24 10:36:42.491497686 +0000 UTC m=+0.152235663 container create 72bc55976cf66fd08460313f50ab55ce7b977a8ca1bf4f8f5a5faeed2f8ae62b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi031, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T10:36:42.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.606+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2ec101f30 con 0x7fe2ec1067f0 2024-03-24T10:36:42.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.607+0000 7fe2deffd700 1 -- 172.21.15.31:0/1235633905 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe2e0026070 con 0x7fe2ec1067f0 2024-03-24T10:36:42.609 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:42.609 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:33:51.503746Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:36:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 msgr2=0x7fe2d803ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 0x7fe2d803ad80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8006fd0 tx=0x7fe2e8006e40 comp rx=0 tx=0).stop 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 msgr2=0x7fe2ec198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005490 tx=0x7fe2e0005570 comp rx=0 tx=0).stop 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 shutdown_connections 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2d80388c0 0x7fe2d803ad80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 --2- 172.21.15.31:0/1235633905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2ec1067f0 0x7fe2ec198270 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 >> 172.21.15.31:0/1235633905 conn(0x7fe2ec0fc300 msgr2=0x7fe2ec107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 shutdown_connections 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:42.609+0000 7fe2f404a700 1 -- 172.21.15.31:0/1235633905 wait complete. 2024-03-24T10:36:42.611 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 1 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 podman[37310]: 2024-03-24 10:36:42.719106375 +0000 UTC m=+0.379844352 container init 72bc55976cf66fd08460313f50ab55ce7b977a8ca1bf4f8f5a5faeed2f8ae62b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi031, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 podman[37310]: 2024-03-24 10:36:42.72258556 +0000 UTC m=+0.383323546 container start 72bc55976cf66fd08460313f50ab55ce7b977a8ca1bf4f8f5a5faeed2f8ae62b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi031, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable), process ceph-mon, pid 2 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: pidfile_write: ignore empty --pid-file 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: load: jerasure load: lrc 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: RocksDB version: 7.9.2 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Git sha 0 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Compile date 2024-03-23 03:05:26 2024-03-24T10:36:42.866 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: DB SUMMARY 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: DB Session ID: 708OGAPQP3D6H7Q5D6BL 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: CURRENT file: CURRENT 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi031/store.db dir, Total Num: 0, files: 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi031/store.db: 000004.log size: 511 ; 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.error_if_exists: 0 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.create_if_missing: 0 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.paranoid_checks: 1 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T10:36:42.867 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.env: 0x557560773720 2024-03-24T10:36:42.868 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T10:36:42.868 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.info_log: 0x557563457380 2024-03-24T10:36:42.868 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T10:36:42.868 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.statistics: (nil) 2024-03-24T10:36:42.869 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.use_fsync: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_log_file_size: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_fallocate: 1 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.use_direct_reads: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T10:36:42.870 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.db_log_dir: 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.wal_dir: 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T10:36:42.871 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.write_buffer_manager: 0x5575626d0140 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T10:36:42.872 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.unordered_write: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.row_cache: None 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.wal_filter: None 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.two_write_queues: 0 2024-03-24T10:36:42.873 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.wal_compression: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.atomic_flush: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.log_readahead_size: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T10:36:42.874 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_background_jobs: 2 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_background_compactions: -1 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_subcompactions: 1 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_open_files: -1 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T10:36:42.875 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_background_flushes: -1 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Compression algorithms supported: 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kZSTD supported: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kXpressCompression supported: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kZlibCompression supported: 1 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kSnappyCompression supported: 1 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kLZ4Compression supported: 1 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: kBZip2Compression supported: 0 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T10:36:42.876 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000005 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.merge_operator: 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_filter: None 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5575634574a0) 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T10:36:42.877 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_type: 0 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_index_type: 0 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_shortening: 1 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: checksum: 4 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: no_block_cache: 0 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache: 0x5575627711f0 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_options: 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: capacity : 536870912 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: num_shard_bits : 4 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: strict_capacity_limit : 0 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-03-24T10:36:42.878 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_cache_compressed: (nil) 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: persistent_cache: (nil) 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size: 4096 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_size_deviation: 10 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_restart_interval: 16 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: index_block_restart_interval: 1 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: metadata_block_size: 4096 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: partition_filters: 0 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: use_delta_encoding: 1 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: filter_policy: bloomfilter 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: whole_key_filtering: 1 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: verify_compression: 0 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: format_version: 5 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_index_compression: 1 2024-03-24T10:36:42.879 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: block_align: 0 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: max_auto_readahead_size: 262144 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: prepopulate_block_cache: 0 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: initial_auto_readahead_size: 8192 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression: NoCompression 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.num_levels: 7 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T10:36:42.880 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T10:36:42.881 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T10:36:42.882 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T10:36:42.883 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T10:36:42.884 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.table_properties_collectors: 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.inplace_update_support: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.bloom_locality: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.max_successive_merges: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T10:36:42.885 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.ttl: 2592000 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enable_blob_files: false 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.min_blob_size: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T10:36:42.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi031/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d2f312a1-47db-4696-a476-57ba9b122577 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276602762230, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276602762686, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711276602, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d2f312a1-47db-4696-a476-57ba9b122577", "db_session_id": "708OGAPQP3D6H7Q5D6BL", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276602762782, "job": 1, "event": "recovery_finished"} 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi031/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55756280e000 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: DB pointer 0x5575627fc000 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** DB Stats ** 2024-03-24T10:36:42.887 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T10:36:42.888 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T10:36:42.889 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Block cache BinnedLRUCache@0x5575627711f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031 does not exist in monmap, will attempt to join an existing cluster 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: using public_addr v2:172.21.15.31:0/0 -> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: starting mon.smithi031 rank -1 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi031 fsid e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(???) e0 preinit fsid e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).mds e1 new map 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).mds e1 print_map 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: e1 2024-03-24T10:36:42.890 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: legacy client fscid: -1 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout: No filesystems configured 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:36:42.891 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: Deploying daemon mon.smithi031 on smithi031 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/1534987538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:42.892 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/1235633905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:42.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 ceph-mon[37345]: mon.smithi031@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-03-24T10:36:42.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 bash[37310]: 72bc55976cf66fd08460313f50ab55ce7b977a8ca1bf4f8f5a5faeed2f8ae62b 2024-03-24T10:36:42.893 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:42 smithi031 systemd[1]: Started Ceph mon.smithi031 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:36:43.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:42 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/1235633905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:44.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T10:36:44.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:36:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:36:48.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:36:48.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: mon.smithi026 calling monitor election 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.ucwncv/crt"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: mon.smithi031 calling monitor election 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: mon.smithi026 is new leader, mons smithi026,smithi031 in quorum (ranks 0,1) 2024-03-24T10:36:48.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: monmap e2: 2 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],smithi031=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: fsmap 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: mgrmap e17: smithi026.bbjcru(active, since 34s) 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: Standby manager daemon smithi031.ucwncv started 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: overall HEALTH_OK 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.ucwncv/key"}]: dispatch 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:36:48.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: mon.smithi026 calling monitor election 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.ucwncv/crt"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: mon.smithi031 calling monitor election 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.203 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: mon.smithi026 is new leader, mons smithi026,smithi031 in quorum (ranks 0,1) 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: monmap e2: 2 mons at {smithi026=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],smithi031=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: fsmap 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: mgrmap e17: smithi026.bbjcru(active, since 34s) 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: Standby manager daemon smithi031.ucwncv started 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: overall HEALTH_OK 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi031.ucwncv/key"}]: dispatch 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.? 172.21.15.31:0/1403080015' entity='mgr.smithi031.ucwncv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:36:48.204 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: mgrmap e18: smithi026.bbjcru(active, since 34s), standbys: smithi031.ucwncv 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi031.ucwncv", "id": "smithi031.ucwncv"}]: dispatch 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:48.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:48.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: mgrmap e18: smithi026.bbjcru(active, since 34s), standbys: smithi031.ucwncv 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr metadata", "who": "smithi031.ucwncv", "id": "smithi031.ucwncv"}]: dispatch 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:49.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/626944148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 msgr2=0x7f0e3c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/626944148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e3c005a50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e54068430 tx=0x7f0e4c00feb0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/626944148 shutdown_connections 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/626944148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e3c005a50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/626944148 >> 172.21.15.31:0/626944148 conn(0x7f0e54075dd0 msgr2=0x7f0e540761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/626944148 shutdown_connections 2024-03-24T10:36:49.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.324+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/626944148 wait complete. 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 Processor -- start 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 -- start start 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e541982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54198e50 con 0x7f0e3c0035c0 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.325+0000 7f0e5c5d8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54198fc0 con 0x7f0e54198830 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:49.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48572/0 (socket says 172.21.15.31:48572) 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 -- 172.21.15.31:0/2415836233 learned_addr learned my addr 172.21.15.31:0/2415836233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5a374700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e541982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 -- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 msgr2=0x7f0e5419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 -- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 msgr2=0x7f0e5419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e59b73700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5a374700 1 -- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 msgr2=0x7f0e5419cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5a374700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5a374700 1 -- 172.21.15.31:0/2415836233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c00f900 con 0x7f0e3c0035c0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5a374700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e541982f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e54068430 tx=0x7f0e4c005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e4c004b90 con 0x7f0e3c0035c0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e4c013e20 con 0x7f0e3c0035c0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.326+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5419d1e0 con 0x7f0e3c0035c0 2024-03-24T10:36:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.327+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e4c011bb0 con 0x7f0e3c0035c0 2024-03-24T10:36:49.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.327+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5419d7b0 con 0x7f0e3c0035c0 2024-03-24T10:36:49.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.327+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e54104c10 con 0x7f0e3c0035c0 2024-03-24T10:36:49.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.328+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f0e4c013960 con 0x7f0e3c0035c0 2024-03-24T10:36:49.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.328+0000 7f0e537fe700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 0x7f0e4006f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:49.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.329+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f0e4c08dbc0 con 0x7f0e3c0035c0 2024-03-24T10:36:49.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.329+0000 7f0e59b73700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 0x7f0e4006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:49.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.329+0000 7f0e59b73700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 0x7f0e4006f220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009180 tx=0x7f0e48008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:49.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.334+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e4c058fd0 con 0x7f0e3c0035c0 2024-03-24T10:36:49.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.629+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e5419da20 con 0x7f0e3c0035c0 2024-03-24T10:36:49.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.629+0000 7f0e537fe700 1 -- 172.21.15.31:0/2415836233 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1018 (secure 0 0 0) 0x7f0e4c031090 con 0x7f0e3c0035c0 2024-03-24T10:36:49.630 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-24T10:36:49.630 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:36:42.788560Z","created":"2024-03-24T10:33:51.503746Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi026","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi031","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 msgr2=0x7f0e4006f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 0x7f0e4006f220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009180 tx=0x7f0e48008040 comp rx=0 tx=0).stop 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 msgr2=0x7f0e541982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e541982f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e54068430 tx=0x7f0e4c005300 comp rx=0 tx=0).stop 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 shutdown_connections 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f0e4006cd60 0x7f0e4006f220 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e3c0035c0 0x7f0e541982f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 --2- 172.21.15.31:0/2415836233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e54198830 0x7f0e5419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 >> 172.21.15.31:0/2415836233 conn(0x7f0e54075dd0 msgr2=0x7f0e54100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 shutdown_connections 2024-03-24T10:36:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:36:49.632+0000 7f0e5c5d8700 1 -- 172.21.15.31:0/2415836233 wait complete. 2024-03-24T10:36:49.633 INFO:teuthology.orchestra.run.smithi031.stderr:dumped monmap epoch 2 2024-03-24T10:36:50.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:49 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/2415836233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:50.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:49 smithi031 ceph-mon[37345]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:50.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:49 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2415836233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:36:50.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:49 smithi026 ceph-mon[29210]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:50.373 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T10:36:50.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T10:36:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.779 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:36:51.779 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:36:51.779 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:51.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:51.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:51.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:36:51.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:52.777 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:52 smithi026 ceph-mon[29210]: Updating smithi031:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:52.777 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:52 smithi026 ceph-mon[29210]: Updating smithi026:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:52.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:52 smithi026 ceph-mon[29210]: Reconfiguring mon.smithi026 (unknown last config time)... 2024-03-24T10:36:52.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:52 smithi026 ceph-mon[29210]: Reconfiguring daemon mon.smithi026 on smithi026 2024-03-24T10:36:52.778 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:52 smithi026 ceph-mon[29210]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:52.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:52 smithi031 ceph-mon[37345]: Updating smithi031:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:52.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:52 smithi031 ceph-mon[37345]: Updating smithi026:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:36:52.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:52 smithi031 ceph-mon[37345]: Reconfiguring mon.smithi026 (unknown last config time)... 2024-03-24T10:36:52.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:52 smithi031 ceph-mon[37345]: Reconfiguring daemon mon.smithi026 on smithi026 2024-03-24T10:36:52.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:52 smithi031 ceph-mon[37345]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:53.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 -- 172.21.15.26:0/1518893280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 msgr2=0x7fb3440ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:53.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/1518893280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb3440ffde0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb33400b0a0 tx=0x7fb33400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 -- 172.21.15.26:0/1518893280 shutdown_connections 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/1518893280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 0x7fb3441047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/1518893280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb3440ffde0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 -- 172.21.15.26:0/1518893280 >> 172.21.15.26:0/1518893280 conn(0x7fb344076640 msgr2=0x7fb3440fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 -- 172.21.15.26:0/1518893280 shutdown_connections 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.275+0000 7fb34b5fe700 1 -- 172.21.15.26:0/1518893280 wait complete. 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 Processor -- start 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 -- start start 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 0x7fb34419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34419dce0 con 0x7fb3440ffa00 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34b5fe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441a1960 con 0x7fb344100320 2024-03-24T10:36:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36954/0 (socket says 172.21.15.26:36954) 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 -- 172.21.15.26:0/4085828760 learned_addr learned my addr 172.21.15.26:0/4085828760 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb348b99700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 0x7fb34419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 -- 172.21.15.26:0/4085828760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 msgr2=0x7fb34419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 0x7fb34419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 -- 172.21.15.26:0/4085828760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334009d00 con 0x7fb3440ffa00 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb34939a700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb334015040 tx=0x7fb334003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.276+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb33400e040 con 0x7fb3440ffa00 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.277+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb334007cd0 con 0x7fb3440ffa00 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.277+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3441a1be0 con 0x7fb3440ffa00 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.277+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb33401d6b0 con 0x7fb3440ffa00 2024-03-24T10:36:53.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.277+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3441a20d0 con 0x7fb3440ffa00 2024-03-24T10:36:53.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.277+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb344068d60 con 0x7fb3440ffa00 2024-03-24T10:36:53.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.278+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7fb334004040 con 0x7fb3440ffa00 2024-03-24T10:36:53.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.278+0000 7fb3427fc700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 0x7fb32c06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:53.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.278+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb33408c8d0 con 0x7fb3440ffa00 2024-03-24T10:36:53.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.278+0000 7fb348b99700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 0x7fb32c06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:53.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.279+0000 7fb348b99700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 0x7fb32c06f120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb34419e5d0 tx=0x7fb338008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:53.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.283+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb334057d80 con 0x7fb3440ffa00 2024-03-24T10:36:53.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.497+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb3441a2340 con 0x7fb3440ffa00 2024-03-24T10:36:53.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.497+0000 7fb3427fc700 1 -- 172.21.15.26:0/4085828760 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7fb33408c090 con 0x7fb3440ffa00 2024-03-24T10:36:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:# minimal ceph.conf for e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:36:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-03-24T10:36:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:36:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: mon_host = [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 msgr2=0x7fb32c06f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 0x7fb32c06f120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb34419e5d0 tx=0x7fb338008040 comp rx=0 tx=0).stop 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 msgr2=0x7fb34419d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb334015040 tx=0x7fb334003ce0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 shutdown_connections 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb32c06cc60 0x7fb32c06f120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3440ffa00 0x7fb34419d0c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 --2- 172.21.15.26:0/4085828760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb344100320 0x7fb34419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 >> 172.21.15.26:0/4085828760 conn(0x7fb344076640 msgr2=0x7fb34410e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 shutdown_connections 2024-03-24T10:36:53.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:36:53.499+0000 7fb34b5fe700 1 -- 172.21.15.26:0/4085828760 wait complete. 2024-03-24T10:36:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: Reconfiguring mgr.smithi026.bbjcru (unknown last config time)... 2024-03-24T10:36:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.bbjcru", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: Reconfiguring daemon mgr.smithi026.bbjcru on smithi026 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/4085828760' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:53 smithi031 ceph-mon[37345]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: Reconfiguring mgr.smithi026.bbjcru (unknown last config time)... 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi026.bbjcru", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: Reconfiguring daemon mgr.smithi026.bbjcru on smithi026 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/4085828760' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:54.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:53 smithi026 ceph-mon[29210]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:54.446 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T10:36:54.446 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:36:54.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T10:36:54.478 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:36:54.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:36:54.547 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:36:54.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T10:36:54.581 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:36:54.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:36:54.652 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T10:36:54.653 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:36:54.653 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:36:54.667 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:36:54.668 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-03-24T10:36:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:36:54.759 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-24 10:36:04.952125475 +0000 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-24 10:29:55.510371533 +0000 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-24 10:29:55.510371533 +0000 2024-03-24T10:36:54.760 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-24 10:29:55.510371533 +0000 2024-03-24T10:36:54.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:36:54.790 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-24T10:36:54.790 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-24T10:36:54.790 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000109094 s, 4.7 MB/s 2024-03-24T10:36:54.791 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:36:54.846 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-24 10:36:04.953125461 +0000 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-24 10:29:55.909365836 +0000 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-24 10:29:55.909365836 +0000 2024-03-24T10:36:54.901 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-24 10:29:55.909365836 +0000 2024-03-24T10:36:54.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:36:54.968 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-24T10:36:54.968 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-24T10:36:54.968 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.67e-05 s, 5.9 MB/s 2024-03-24T10:36:54.969 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:36:55.026 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-24 10:36:04.953125461 +0000 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-24 10:29:56.254360911 +0000 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-24 10:29:56.254360911 +0000 2024-03-24T10:36:55.081 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-24 10:29:56.254360911 +0000 2024-03-24T10:36:55.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:36:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-24T10:36:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-24T10:36:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.0105e-05 s, 5.7 MB/s 2024-03-24T10:36:55.149 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:36:55.204 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-03-24T10:36:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:36:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:55.261 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T10:36:55.261 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:55.262 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:55.262 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-24 10:36:04.953125461 +0000 2024-03-24T10:36:55.262 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-24 10:29:56.588356142 +0000 2024-03-24T10:36:55.262 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-24 10:29:56.588356142 +0000 2024-03-24T10:36:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-24 10:29:56.588356142 +0000 2024-03-24T10:36:55.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:36:55.328 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-24T10:36:55.328 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-24T10:36:55.328 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.8312e-05 s, 5.2 MB/s 2024-03-24T10:36:55.330 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:36:55.386 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:36:55.386 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:36:55.404 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:36:55.404 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-03-24T10:36:55.461 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-24 10:36:40.883940581 +0000 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-24 10:29:55.190029276 +0000 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-24 10:29:55.190029276 +0000 2024-03-24T10:36:55.462 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-24 10:29:55.190029276 +0000 2024-03-24T10:36:55.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:36:55.531 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-24T10:36:55.531 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-24T10:36:55.531 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000205889 s, 2.5 MB/s 2024-03-24T10:36:55.532 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:36:55.590 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-03-24T10:36:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-24 10:36:40.884940566 +0000 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-24 10:29:55.514024384 +0000 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-24 10:29:55.514024384 +0000 2024-03-24T10:36:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-24 10:29:55.514024384 +0000 2024-03-24T10:36:55.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:36:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-24T10:36:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-24T10:36:55.715 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.0002243 s, 2.3 MB/s 2024-03-24T10:36:55.717 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:36:55.775 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-24 10:36:40.884940566 +0000 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-24 10:29:55.897018602 +0000 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-24 10:29:55.897018602 +0000 2024-03-24T10:36:55.831 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-24 10:29:55.897018602 +0000 2024-03-24T10:36:55.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:36:55.897 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-24T10:36:55.897 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-24T10:36:55.897 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000196338 s, 2.6 MB/s 2024-03-24T10:36:55.899 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:36:55.956 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-24 10:36:40.884940566 +0000 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-24 10:29:56.259013138 +0000 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-24 10:29:56.259013138 +0000 2024-03-24T10:36:56.011 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-24 10:29:56.259013138 +0000 2024-03-24T10:36:56.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:36:56.074 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:56.074 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:56.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: Reconfiguring ceph-exporter.smithi026 (monmap changed)... 2024-03-24T10:36:56.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi026", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:36:56.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:56.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: Reconfiguring daemon ceph-exporter.smithi026 on smithi026 2024-03-24T10:36:56.075 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:55 smithi031 ceph-mon[37345]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-24T10:36:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-24T10:36:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000193882 s, 2.6 MB/s 2024-03-24T10:36:56.078 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:36:56.134 INFO:tasks.cephadm:Deploying osd.0 on smithi026 with /dev/vg_nvme/lv_4... 2024-03-24T10:36:56.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: Reconfiguring ceph-exporter.smithi026 (monmap changed)... 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi026", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: Reconfiguring daemon ceph-exporter.smithi026 on smithi026 2024-03-24T10:36:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:55 smithi026 ceph-mon[29210]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:56.452 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:36:57.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:57.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:57.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: Reconfiguring crash.smithi026 (monmap changed)... 2024-03-24T10:36:57.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:36:57.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:57.530 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:57 smithi026 ceph-mon[29210]: Reconfiguring daemon crash.smithi026 on smithi026 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: Reconfiguring crash.smithi026 (monmap changed)... 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi026", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:57.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:57 smithi031 ceph-mon[37345]: Reconfiguring daemon crash.smithi026 on smithi026 2024-03-24T10:36:58.535 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:58 smithi026 ceph-mon[29210]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:58.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:58 smithi031 ceph-mon[37345]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832523 s, 1.3 GB/s 2024-03-24T10:36:59.400 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-24T10:36:59.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:59 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:59.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:59 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:59.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:59 smithi026 ceph-mon[29210]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-03-24T10:36:59.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:59 smithi026 ceph-mon[29210]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-03-24T10:36:59.928 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:36:59 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:36:59.949 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_4 2024-03-24T10:36:59.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:59 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:59.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:59 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:36:59.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:59 smithi031 ceph-mon[37345]: Reconfiguring alertmanager.smithi026 (dependencies changed)... 2024-03-24T10:36:59.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:59 smithi031 ceph-mon[37345]: Reconfiguring daemon alertmanager.smithi026 on smithi026 2024-03-24T10:36:59.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:36:59 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:00.815 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:00 smithi026 ceph-mon[29210]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:00.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:00 smithi031 ceph-mon[37345]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:37:02.015 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:01 smithi026 ceph-mon[29210]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:02.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:01 smithi031 ceph-mon[37345]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 -- 172.21.15.26:0/3719263571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 msgr2=0x7f8c6c103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 --2- 172.21.15.26:0/3719263571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c103950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009910 tx=0x7f8c68009c20 comp rx=0 tx=0).stop 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 -- 172.21.15.26:0/3719263571 shutdown_connections 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 --2- 172.21.15.26:0/3719263571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 0x7f8c6c106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 --2- 172.21.15.26:0/3719263571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c103950 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 -- 172.21.15.26:0/3719263571 >> 172.21.15.26:0/3719263571 conn(0x7f8c6c0faf70 msgr2=0x7f8c6c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 -- 172.21.15.26:0/3719263571 shutdown_connections 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.776+0000 7f8c7387d700 1 -- 172.21.15.26:0/3719263571 wait complete. 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 Processor -- start 2024-03-24T10:37:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 -- start start 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 0x7f8c6c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c196ed0 con 0x7f8c6c101560 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c7387d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c197010 con 0x7f8c6c103e90 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54814/0 (socket says 172.21.15.26:54814) 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 -- 172.21.15.26:0/1249934915 learned_addr learned my addr 172.21.15.26:0/1249934915 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c70e18700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 0x7f8c6c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 -- 172.21.15.26:0/1249934915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 msgr2=0x7f8c6c1968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 0x7f8c6c1968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 -- 172.21.15.26:0/1249934915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680095f0 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.777+0000 7f8c71619700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006010 tx=0x7f8c68004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.778+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c6801d070 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.778+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c6800beb0 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.778+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c19ba10 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.778+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c6800f6d0 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.778+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c19bf80 con 0x7f8c6c101560 2024-03-24T10:37:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.779+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c04ea80 con 0x7f8c6c101560 2024-03-24T10:37:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.779+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f8c68022ac0 con 0x7f8c6c101560 2024-03-24T10:37:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.780+0000 7f8c627fc700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 0x7f8c5806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.780+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8c6808cd10 con 0x7f8c6c101560 2024-03-24T10:37:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.780+0000 7f8c70e18700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 0x7f8c5806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.780+0000 7f8c70e18700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 0x7f8c5806f110 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c006fd0 tx=0x7f8c5c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:03.784+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c6805be20 con 0x7f8c6c101560 2024-03-24T10:37:03.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:03 smithi026 ceph-mon[29210]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:04.025+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c6c104730 con 0x7f8c5806cc50 2024-03-24T10:37:04.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:03 smithi031 ceph-mon[37345]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:05.090 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:04 smithi026 ceph-mon[29210]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:05.090 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:04 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:37:05.090 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:04 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:37:05.090 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:04 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:05.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:04 smithi031 ceph-mon[37345]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:05.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:04 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:37:05.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:04 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:37:05.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:04 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:06.270 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:06 smithi026 ceph-mon[29210]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:06 smithi031 ceph-mon[37345]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:07.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:07 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:07.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:07 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:07.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:07 smithi026 ceph-mon[29210]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-03-24T10:37:07.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:07 smithi026 ceph-mon[29210]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-03-24T10:37:07.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:07 smithi026 ceph-mon[29210]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:07 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:07 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:07 smithi031 ceph-mon[37345]: Reconfiguring grafana.smithi026 (dependencies changed)... 2024-03-24T10:37:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:07 smithi031 ceph-mon[37345]: Reconfiguring daemon grafana.smithi026 on smithi026 2024-03-24T10:37:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:07 smithi031 ceph-mon[37345]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:08.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:08 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1068566509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20a15cb6-4489-4794-a110-50a62c92cc96"}]: dispatch 2024-03-24T10:37:08.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:08 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1068566509' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20a15cb6-4489-4794-a110-50a62c92cc96"}]': finished 2024-03-24T10:37:08.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:08 smithi026 ceph-mon[29210]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T10:37:08.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:08 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:08.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:08 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3493380466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:08 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1068566509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20a15cb6-4489-4794-a110-50a62c92cc96"}]: dispatch 2024-03-24T10:37:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:08 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1068566509' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20a15cb6-4489-4794-a110-50a62c92cc96"}]': finished 2024-03-24T10:37:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:08 smithi031 ceph-mon[37345]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T10:37:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:08 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:08 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3493380466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:10.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:09 smithi031 ceph-mon[37345]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:10.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:09 smithi026 ceph-mon[29210]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:11.882 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:11 smithi026 ceph-mon[29210]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:12.202 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:11 smithi031 ceph-mon[37345]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:13.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:13 smithi026 ceph-mon[29210]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:13.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:14.001 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:13 smithi031 ceph-mon[37345]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:14.002 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:16.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:16 smithi031 ceph-mon[37345]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:16.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:16 smithi026 ceph-mon[29210]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:17.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:17.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:17.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:17 smithi026 ceph-mon[29210]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-03-24T10:37:17.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:17 smithi026 ceph-mon[29210]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-03-24T10:37:17.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:17 smithi026 ceph-mon[29210]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:17 smithi031 ceph-mon[37345]: Reconfiguring prometheus.smithi026 (dependencies changed)... 2024-03-24T10:37:18.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:17 smithi031 ceph-mon[37345]: Reconfiguring daemon prometheus.smithi026 on smithi026 2024-03-24T10:37:18.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:17 smithi031 ceph-mon[37345]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:19.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:18 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T10:37:19.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:18 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:19.198 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:18 smithi026 ceph-mon[29210]: Deploying daemon osd.0 on smithi026 2024-03-24T10:37:19.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:18 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T10:37:19.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:18 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:19.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:18 smithi031 ceph-mon[37345]: Deploying daemon osd.0 on smithi026 2024-03-24T10:37:20.096 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:19 smithi026 ceph-mon[29210]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:20.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:19 smithi031 ceph-mon[37345]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:21.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:21 smithi026 ceph-mon[29210]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:22.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:21 smithi031 ceph-mon[37345]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: Reconfiguring ceph-exporter.smithi031 (monmap changed)... 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi031", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:23.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:23 smithi026 ceph-mon[29210]: Reconfiguring daemon ceph-exporter.smithi031 on smithi031 2024-03-24T10:37:23.601 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:23.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:23.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: Reconfiguring ceph-exporter.smithi031 (monmap changed)... 2024-03-24T10:37:23.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi031", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T10:37:23.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:23.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:23 smithi031 ceph-mon[37345]: Reconfiguring daemon ceph-exporter.smithi031 on smithi031 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:37:24.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:24.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:24.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:24.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:24.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: Reconfiguring crash.smithi031 (monmap changed)... 2024-03-24T10:37:24.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi031", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T10:37:24.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.ucwncv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:37:25.843 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:25 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: Reconfiguring daemon crash.smithi031 on smithi031 2024-03-24T10:37:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi031.ucwncv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:37:25.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:37:25.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:25 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: Reconfiguring mgr.smithi031.ucwncv (monmap changed)... 2024-03-24T10:37:26.760 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: Reconfiguring daemon mgr.smithi031.ucwncv on smithi031 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:37:26.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: Reconfiguring mgr.smithi031.ucwncv (monmap changed)... 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: Reconfiguring daemon mgr.smithi031.ucwncv on smithi031 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:37:26.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:27.909 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:27 smithi026 ceph-mon[29210]: Reconfiguring mon.smithi031 (monmap changed)... 2024-03-24T10:37:27.909 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:27 smithi026 ceph-mon[29210]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-24T10:37:27.909 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:27.909 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:27.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:27 smithi031 ceph-mon[37345]: Reconfiguring mon.smithi031 (monmap changed)... 2024-03-24T10:37:27.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:27 smithi031 ceph-mon[37345]: Reconfiguring daemon mon.smithi031 on smithi031 2024-03-24T10:37:27.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:27.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:28.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:28.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:29.936 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T10:37:29.936 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T10:37:29.936 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T10:37:29.936 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T10:37:29.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T10:37:29.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T10:37:29.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi026.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi026.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi026.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T10:37:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:30.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.478+0000 7f8c627fc700 1 -- 172.21.15.26:0/1249934915 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c6c104730 con 0x7f8c5806cc50 2024-03-24T10:37:30.480 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 0 on host 'smithi026' 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 msgr2=0x7f8c5806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 0x7f8c5806f110 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c006fd0 tx=0x7f8c5c008040 comp rx=0 tx=0).stop 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 msgr2=0x7f8c6c196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006010 tx=0x7f8c68004970 comp rx=0 tx=0).stop 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 shutdown_connections 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8c5806cc50 0x7f8c5806f110 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c6c101560 0x7f8c6c196370 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 --2- 172.21.15.26:0/1249934915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c6c103e90 0x7f8c6c1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 >> 172.21.15.26:0/1249934915 conn(0x7f8c6c0faf70 msgr2=0x7f8c6c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 shutdown_connections 2024-03-24T10:37:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:30.480+0000 7f8c7387d700 1 -- 172.21.15.26:0/1249934915 wait complete. 2024-03-24T10:37:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:30 smithi026 ceph-mon[29210]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:30 smithi031 ceph-mon[37345]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:30.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:30.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:31.077 DEBUG:teuthology.orchestra.run.smithi026:osd.0> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T10:37:31.080 INFO:tasks.cephadm:Deploying osd.1 on smithi026 with /dev/vg_nvme/lv_3... 2024-03-24T10:37:31.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T10:37:31.346 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:37:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:37:31.232+0000 7fce37a27640 -1 osd.0 0 log_to_monitors true 2024-03-24T10:37:31.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:31 smithi031 ceph-mon[37345]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T10:37:31.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:31 smithi026 ceph-mon[29210]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T10:37:32.507 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:37:32.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:37:32.673+0000 7fce2c88a700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T10:37:32.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:37:32.676+0000 7fce26e7c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:37:32.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-mon[29210]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:32.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-mon[29210]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T10:37:32.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-mon[29210]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T10:37:32.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:32.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:32 smithi026 ceph-mon[29210]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:37:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:32 smithi031 ceph-mon[37345]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:32 smithi031 ceph-mon[37345]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T10:37:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:32 smithi031 ceph-mon[37345]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T10:37:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:32 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:32 smithi031 ceph-mon[37345]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:37:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:33 smithi026 ceph-mon[29210]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:37:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:33 smithi026 ceph-mon[29210]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T10:37:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:33 smithi031 ceph-mon[37345]: from='osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:37:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:33 smithi031 ceph-mon[37345]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T10:37:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] boot 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T10:37:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: osd.0 [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] boot 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T10:37:34.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00917313 s, 1.1 GB/s 2024-03-24T10:37:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-24T10:37:35.726 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:35 smithi026 ceph-mon[29210]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T10:37:35.726 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:35 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:35.726 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:35 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:35.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:35 smithi031 ceph-mon[37345]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T10:37:35.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:35 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:35.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:35 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:36.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_3 2024-03-24T10:37:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:36 smithi031 ceph-mon[37345]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:36.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:36 smithi026 ceph-mon[29210]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:37:38.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:38 smithi031 ceph-mon[37345]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:38.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:38 smithi026 ceph-mon[29210]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:40.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.236+0000 7f15afd00700 1 -- 172.21.15.26:0/642353428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a80ff160 msgr2=0x7f15a80ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.236+0000 7f15afd00700 1 --2- 172.21.15.26:0/642353428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a80ff160 0x7f15a80ff580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f159800b0a0 tx=0x7f159800b3b0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- 172.21.15.26:0/642353428 shutdown_connections 2024-03-24T10:37:40.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 --2- 172.21.15.26:0/642353428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a8100360 0x7f15a81007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 --2- 172.21.15.26:0/642353428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a80ff160 0x7f15a80ff580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- 172.21.15.26:0/642353428 >> 172.21.15.26:0/642353428 conn(0x7f15a80fa700 msgr2=0x7f15a80fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:40.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- 172.21.15.26:0/642353428 shutdown_connections 2024-03-24T10:37:40.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- 172.21.15.26:0/642353428 wait complete. 2024-03-24T10:37:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 Processor -- start 2024-03-24T10:37:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- start start 2024-03-24T10:37:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a80ff160 0x7f15a8196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a8197090 con 0x7f15a8100360 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.237+0000 7f15afd00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a81971d0 con 0x7f15a80ff160 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ada9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a80ff160 0x7f15a8196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45364/0 (socket says 172.21.15.26:45364) 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 -- 172.21.15.26:0/1417786136 learned_addr learned my addr 172.21.15.26:0/1417786136 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 -- 172.21.15.26:0/1417786136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a80ff160 msgr2=0x7f15a8196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a80ff160 0x7f15a8196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 -- 172.21.15.26:0/1417786136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598009d00 con 0x7f15a8100360 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15ad29b700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f159c00dd40 tx=0x7f159c00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f159c00ad10 con 0x7f15a8100360 2024-03-24T10:37:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f159c00ae70 con 0x7f15a8100360 2024-03-24T10:37:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f159c0168a0 con 0x7f15a8100360 2024-03-24T10:37:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a819bc30 con 0x7f15a8100360 2024-03-24T10:37:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.238+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a819c1b0 con 0x7f15a8100360 2024-03-24T10:37:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.239+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a818e450 con 0x7f15a8100360 2024-03-24T10:37:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.240+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f159c00e450 con 0x7f15a8100360 2024-03-24T10:37:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.240+0000 7f15a6ffd700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 0x7f159006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.240+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f159c08b2d0 con 0x7f15a8100360 2024-03-24T10:37:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.240+0000 7f15ada9c700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 0x7f159006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.241+0000 7f15ada9c700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 0x7f159006f120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1598009fd0 tx=0x7f1598006000 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:40.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.245+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f159c05a060 con 0x7f15a8100360 2024-03-24T10:37:40.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:37:40.460+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f15a80611f0 con 0x7f159006cc60 2024-03-24T10:37:40.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:40.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:40 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:37:40.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:40 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:41.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:41 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi026 2024-03-24T10:37:41.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:41 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi026 to 11845M 2024-03-24T10:37:41.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:41 smithi031 ceph-mon[37345]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:41.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:41 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi026 2024-03-24T10:37:41.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:41 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi026 to 11845M 2024-03-24T10:37:41.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:41 smithi026 ceph-mon[29210]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:42.821 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:42 smithi026 ceph-mon[29210]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:42.821 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:42 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:42.821 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:42 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:42.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:42 smithi031 ceph-mon[37345]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:42.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:42.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:43 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3906044498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1ad8eb2-18e8-4012-93f5-417d61a2832e"}]: dispatch 2024-03-24T10:37:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:43 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3906044498' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1ad8eb2-18e8-4012-93f5-417d61a2832e"}]': finished 2024-03-24T10:37:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:43 smithi031 ceph-mon[37345]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T10:37:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:37:43.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:43 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3906044498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1ad8eb2-18e8-4012-93f5-417d61a2832e"}]: dispatch 2024-03-24T10:37:43.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:43 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3906044498' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1ad8eb2-18e8-4012-93f5-417d61a2832e"}]': finished 2024-03-24T10:37:43.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:43 smithi026 ceph-mon[29210]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T10:37:43.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:37:44.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:44 smithi031 ceph-mon[37345]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:44.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:44 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:44.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:44 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1085255922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:44.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:44 smithi026 ceph-mon[29210]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:44.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:44 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:44.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:44 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1085255922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:46.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:46 smithi031 ceph-mon[37345]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:46.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:46.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:46.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:46 smithi026 ceph-mon[29210]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:48.113 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:47 smithi026 ceph-mon[29210]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:48.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:47 smithi031 ceph-mon[37345]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:49.841 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:49 smithi026 ceph-mon[29210]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:50.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:49 smithi031 ceph-mon[37345]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:51.369 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T10:37:51.369 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:51.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T10:37:51.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:52.308 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:52 smithi026 ceph-mon[29210]: Deploying daemon osd.1 on smithi026 2024-03-24T10:37:52.308 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:52 smithi026 ceph-mon[29210]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:52.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:52 smithi031 ceph-mon[37345]: Deploying daemon osd.1 on smithi026 2024-03-24T10:37:52.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:52 smithi031 ceph-mon[37345]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:53.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:53 smithi026 ceph-mon[29210]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:53 smithi031 ceph-mon[37345]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:56.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:56 smithi031 ceph-mon[37345]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:56.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:56 smithi026 ceph-mon[29210]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:57.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:57.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:57.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:57 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:57.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:57 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:58.387 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:58 smithi026 ceph-mon[29210]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:58.387 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:58.387 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:58.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:58 smithi031 ceph-mon[37345]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:37:58.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:58.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:37:59.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:37:59 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:37:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:37:59 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:00.215 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 1 on host 'smithi026' 2024-03-24T10:38:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.215+0000 7f15a6ffd700 1 -- 172.21.15.26:0/1417786136 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f15a80611f0 con 0x7f159006cc60 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 msgr2=0x7f159006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 0x7f159006f120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1598009fd0 tx=0x7f1598006000 comp rx=0 tx=0).stop 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 msgr2=0x7f15a8196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f159c00dd40 tx=0x7f159c00a530 comp rx=0 tx=0).stop 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 shutdown_connections 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f159006cc60 0x7f159006f120 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a80ff160 0x7f15a8196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 --2- 172.21.15.26:0/1417786136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15a8100360 0x7f15a8196a70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 >> 172.21.15.26:0/1417786136 conn(0x7f15a80fa700 msgr2=0x7f15a8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 shutdown_connections 2024-03-24T10:38:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:00.217+0000 7f15afd00700 1 -- 172.21.15.26:0/1417786136 wait complete. 2024-03-24T10:38:00.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:00 smithi031 ceph-mon[37345]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:38:00.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:00 smithi031 ceph-mon[37345]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:38:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:00 smithi026 ceph-mon[29210]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:38:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:00 smithi026 ceph-mon[29210]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:38:00.831 DEBUG:teuthology.orchestra.run.smithi026:osd.1> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T10:38:00.834 INFO:tasks.cephadm:Deploying osd.2 on smithi026 with /dev/vg_nvme/lv_2... 2024-03-24T10:38:00.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:01 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:38:01.138+0000 7f19b9a30700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T10:38:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:38:01.141+0000 7f19b6026700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:38:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:38:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:02 smithi026 ceph-mon[29210]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:38:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:02 smithi026 ceph-mon[29210]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T10:38:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:02 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:02 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:02 smithi026 ceph-mon[29210]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:38:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:02 smithi031 ceph-mon[37345]: from='osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:38:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:02 smithi031 ceph-mon[37345]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T10:38:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:02 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:02 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:02.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:02 smithi031 ceph-mon[37345]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:38:03.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:03 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:38:03.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:03 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:38:03.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:03 smithi026 ceph-mon[29210]: osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] boot 2024-03-24T10:38:03.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:03 smithi026 ceph-mon[29210]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T10:38:03.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:03 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:03 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:38:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:03 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:38:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:03 smithi031 ceph-mon[37345]: osd.1 [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] boot 2024-03-24T10:38:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:03 smithi031 ceph-mon[37345]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T10:38:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:03 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0102887 s, 1.0 GB/s 2024-03-24T10:38:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-24T10:38:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:04 smithi031 ceph-mon[37345]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T10:38:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:04 smithi031 ceph-mon[37345]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:04 smithi026 ceph-mon[29210]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T10:38:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:04 smithi026 ceph-mon[29210]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:04.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_2 2024-03-24T10:38:04.920 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:38:06.122 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:06.123 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:06.123 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:05 smithi026 ceph-mon[29210]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:06.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:06.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:06.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:05 smithi031 ceph-mon[37345]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 -- 172.21.15.26:0/2977530225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ff350 msgr2=0x7f9ff80ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/2977530225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ff350 0x7f9ff80ff770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b0a0 tx=0x7f9fe800b3b0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 -- 172.21.15.26:0/2977530225 shutdown_connections 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/2977530225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/2977530225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ff350 0x7f9ff80ff770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 -- 172.21.15.26:0/2977530225 >> 172.21.15.26:0/2977530225 conn(0x7f9ff80faf70 msgr2=0x7f9ff80fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 -- 172.21.15.26:0/2977530225 shutdown_connections 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.338+0000 7f9fff5e4700 1 -- 172.21.15.26:0/2977530225 wait complete. 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 Processor -- start 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 -- start start 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ff350 0x7f9ff8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81991e0 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9fff5e4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8199320 con 0x7f9ff80ff350 2024-03-24T10:38:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35252/0 (socket says 172.21.15.26:35252) 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 -- 172.21.15.26:0/1492321265 learned_addr learned my addr 172.21.15.26:0/1492321265 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffd380700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ff350 0x7f9ff8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 -- 172.21.15.26:0/1492321265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ff350 msgr2=0x7f9ff8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ff350 0x7f9ff8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 -- 172.21.15.26:0/1492321265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe8009d00 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ffcb7f700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00d8d0 tx=0x7f9fec00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.339+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec0098a0 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.340+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff819dd80 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.340+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9fec010460 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.340+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec00f5f0 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.340+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff819e350 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.340+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff8192840 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.341+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f9fec009a00 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.341+0000 7f9ff67fc700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 0x7f9fe006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.342+0000 7f9ffd380700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 0x7f9fe006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.342+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f9fec08b3e0 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.342+0000 7f9ffd380700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 0x7f9fe006f120 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007820 tx=0x7f9fe8006000 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.346+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9fec059f10 con 0x7f9ff80ffcb0 2024-03-24T10:38:07.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:07.562+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ff80611f0 con 0x7f9fe006cc60 2024-03-24T10:38:07.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:07 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:07.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:07 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:07.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:07 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:07.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:07 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:07.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:07 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:07.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:07 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:08.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:08 smithi026 ceph-mon[29210]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:08.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:08 smithi026 ceph-mon[29210]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:08.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:08 smithi031 ceph-mon[37345]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:08.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:08 smithi031 ceph-mon[37345]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi026 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:10.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi026 to 5922M 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:10.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:10.918 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi026 2024-03-24T10:38:10.918 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.918 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.918 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:10.918 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi026 to 5922M 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:10.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:11.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:11 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/747400248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb"}]: dispatch 2024-03-24T10:38:11.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:11 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/747400248' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb"}]': finished 2024-03-24T10:38:11.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:11 smithi026 ceph-mon[29210]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T10:38:11.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:11 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:11.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:11 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/747400248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb"}]: dispatch 2024-03-24T10:38:11.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:11 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/747400248' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb"}]': finished 2024-03-24T10:38:11.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:11 smithi031 ceph-mon[37345]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T10:38:11.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:11 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:12.621 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:12 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:12.621 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:12 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:12.621 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:12 smithi026 ceph-mon[29210]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:12.621 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:12 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2750775776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:38:12.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:12 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:12.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:12 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:12.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:12 smithi031 ceph-mon[37345]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:12.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:12 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2750775776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:38:14.001 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:13 smithi026 ceph-mon[29210]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:14.001 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:13 smithi031 ceph-mon[37345]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:15.383 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:15 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:15.383 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:15 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:15.383 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:15 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:15.383 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:15 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:15.383 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:15 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:15.397 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:15.398 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:15.398 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:15.398 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:15.398 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:15 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:16.425 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:16 smithi026 ceph-mon[29210]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:16.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:16 smithi031 ceph-mon[37345]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:17.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:17 smithi026 ceph-mon[29210]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:17 smithi031 ceph-mon[37345]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:19.038 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:18 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T10:38:19.038 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:18 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:19.038 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:18 smithi026 ceph-mon[29210]: Deploying daemon osd.2 on smithi026 2024-03-24T10:38:19.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:18 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T10:38:19.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:18 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:19.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:18 smithi031 ceph-mon[37345]: Deploying daemon osd.2 on smithi026 2024-03-24T10:38:20.107 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:19 smithi026 ceph-mon[29210]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:20.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:19 smithi031 ceph-mon[37345]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:22.029 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:21 smithi026 ceph-mon[29210]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:22.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:21 smithi031 ceph-mon[37345]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:23.850 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:23 smithi026 ceph-mon[29210]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:24.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:23 smithi031 ceph-mon[37345]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:25.036 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:25.036 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:25.037 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:25.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:25.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:25.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:26.406 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:26 smithi031 ceph-mon[37345]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:26 smithi026 ceph-mon[29210]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:27.669 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 2 on host 'smithi026' 2024-03-24T10:38:27.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.668+0000 7f9ff67fc700 1 -- 172.21.15.26:0/1492321265 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9ff80611f0 con 0x7f9fe006cc60 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 msgr2=0x7f9fe006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 0x7f9fe006f120 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007820 tx=0x7f9fe8006000 comp rx=0 tx=0).stop 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 msgr2=0x7f9ff8198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00d8d0 tx=0x7f9fec00dbe0 comp rx=0 tx=0).stop 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 shutdown_connections 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9fe006cc60 0x7f9fe006f120 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff80ff350 0x7f9ff8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 --2- 172.21.15.26:0/1492321265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ff80ffcb0 0x7f9ff8198bc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 >> 172.21.15.26:0/1492321265 conn(0x7f9ff80faf70 msgr2=0x7f9ff8107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 shutdown_connections 2024-03-24T10:38:27.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:38:27.671+0000 7f9fff5e4700 1 -- 172.21.15.26:0/1492321265 wait complete. 2024-03-24T10:38:28.285 DEBUG:teuthology.orchestra.run.smithi026:osd.2> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T10:38:28.289 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_4... 2024-03-24T10:38:28.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T10:38:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:38:28.914 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:28.914 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:28 smithi026 ceph-mon[29210]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:28.914 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:28.914 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:28 smithi026 ceph-mon[29210]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:38:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:28 smithi031 ceph-mon[37345]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:28.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:28 smithi031 ceph-mon[37345]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:38:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:29 smithi031 ceph-mon[37345]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T10:38:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:29 smithi031 ceph-mon[37345]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T10:38:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:29.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:29 smithi031 ceph-mon[37345]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-mon[29210]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-mon[29210]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-mon[29210]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-24T10:38:29.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:38:29.670+0000 7fee224e4700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T10:38:29.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:38:29.675+0000 7fee1dad8700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:38:30.213 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T10:38:30.214 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T10:38:30.214 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-24T10:38:30.214 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-24T10:38:30.214 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109718 s, 956 MB/s 2024-03-24T10:38:30.214 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:30.780 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:30.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: from='osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:30.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:30.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:31.010 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:38:31.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:31 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:31.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:31 smithi031 ceph-mon[37345]: osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] boot 2024-03-24T10:38:31.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:31 smithi031 ceph-mon[37345]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T10:38:31.786 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:31 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:31.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:31 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:31.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:31 smithi026 ceph-mon[29210]: osd.2 [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] boot 2024-03-24T10:38:31.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:31 smithi026 ceph-mon[29210]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T10:38:31.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:31 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- 172.21.15.31:0/3439791384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754101ce0 msgr2=0x7fe754102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 --2- 172.21.15.31:0/3439791384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754101ce0 0x7fe754102140 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe740009910 tx=0x7fe740009c20 comp rx=0 tx=0).stop 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- 172.21.15.31:0/3439791384 shutdown_connections 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 --2- 172.21.15.31:0/3439791384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754101ce0 0x7fe754102140 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 --2- 172.21.15.31:0/3439791384 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754100ae0 0x7fe754100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- 172.21.15.31:0/3439791384 >> 172.21.15.31:0/3439791384 conn(0x7fe7540fc0a0 msgr2=0x7fe7540fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- 172.21.15.31:0/3439791384 shutdown_connections 2024-03-24T10:38:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- 172.21.15.31:0/3439791384 wait complete. 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 Processor -- start 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 -- start start 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.434+0000 7fe75aae9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754101ce0 0x7fe7540731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe75aae9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540736e0 con 0x7fe754101ce0 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe75aae9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe754073820 con 0x7fe754100ae0 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49806/0 (socket says 172.21.15.31:49806) 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 -- 172.21.15.31:0/2010804009 learned_addr learned my addr 172.21.15.31:0/2010804009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe753fff700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754101ce0 0x7fe7540731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 -- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754101ce0 msgr2=0x7fe7540731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754101ce0 0x7fe7540731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 -- 172.21.15.31:0/2010804009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400095f0 con 0x7fe754100ae0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe758885700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe74800b480 tx=0x7fe74800b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe748011800 con 0x7fe754100ae0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe748011e20 con 0x7fe754100ae0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe754073b00 con 0x7fe754100ae0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.435+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe748010ea0 con 0x7fe754100ae0 2024-03-24T10:38:32.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.436+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75419e660 con 0x7fe754100ae0 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.436+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe754190700 con 0x7fe754100ae0 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.437+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7fe748010380 con 0x7fe754100ae0 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.437+0000 7fe751ffb700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 0x7fe74406f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.437+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7fe74808b1d0 con 0x7fe754100ae0 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.438+0000 7fe753fff700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 0x7fe74406f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.438+0000 7fe753fff700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 0x7fe74406f120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe740005090 tx=0x7fe740005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:32.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.442+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe748059c00 con 0x7fe754100ae0 2024-03-24T10:38:32.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:32.654+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7540611f0 con 0x7fe74406cc60 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:32.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:32 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:32.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:32 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T10:38:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T10:38:33.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:33 smithi031 ceph-mon[37345]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:38:33.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:33 smithi026 ceph-mon[29210]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T10:38:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67918]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:38:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67918]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67918]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67918]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:34.822 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:38:34.822 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:34.822 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:34.822 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[67974]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.823 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:34.824 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:34.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:35.091 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[68055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:38:35.091 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[68055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:35.092 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[68055]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:35.092 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:34 smithi026 sudo[68055]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:35.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:35.151+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mon.1 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fe748056540 con 0x7fe754100ae0 2024-03-24T10:38:35.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68075]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:38:35.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:35.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68075]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:35.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68075]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:35.625 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68120]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:38:35.625 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68120]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:35.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68120]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:35.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68120]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:35.881 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68182]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:38:35.881 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68182]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:35.881 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68182]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:35.881 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68182]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:36.153 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:38:36.153 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:36.153 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68208]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:36.153 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:35 smithi026 sudo[68208]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 sudo[68231]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:38:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 sudo[68231]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 sudo[68231]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 sudo[68231]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi026 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi026 to 3948M 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/1952054541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]: dispatch 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]: dispatch 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]': finished 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: mgrmap e19: smithi026.bbjcru(active, since 2m), standbys: smithi031.ucwncv 2024-03-24T10:38:36.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:36 smithi026 ceph-mon[29210]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:38:36.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:38:36.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:36.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:36.684 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41430]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41620]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:38:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:38:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41620]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:38:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 sudo[41620]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:38:36.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi026 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi026 to 3948M 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/1952054541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]: dispatch 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]: dispatch 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd6ae4bc-dd6b-4dbc-ac84-b10495658433"}]': finished 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: mgrmap e19: smithi026.bbjcru(active, since 2m), standbys: smithi031.ucwncv 2024-03-24T10:38:36.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:36 smithi031 ceph-mon[37345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:38:37.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:37.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/552777323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:38:37.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: Cluster is now healthy 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:37 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:37.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/552777323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi026"}]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mon metadata", "id": "smithi031"}]: dispatch 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: Cluster is now healthy 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:38:37.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:37 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:38.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:38 smithi031 ceph-mon[37345]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:38.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:38 smithi026 ceph-mon[29210]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:39.601 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:39.601 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:39.601 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:39.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:39.602 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:39.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:40.758 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:40 smithi031 ceph-mon[37345]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:40 smithi026 ceph-mon[29210]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:41.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:41 smithi031 ceph-mon[37345]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:41.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:41 smithi026 ceph-mon[29210]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:43.092 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T10:38:43.092 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:42 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:43.092 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:42 smithi031 ceph-mon[37345]: Deploying daemon osd.3 on smithi031 2024-03-24T10:38:43.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:42 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T10:38:43.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:42 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:43.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:42 smithi026 ceph-mon[29210]: Deploying daemon osd.3 on smithi031 2024-03-24T10:38:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:43 smithi031 ceph-mon[37345]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:44.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:43 smithi026 ceph-mon[29210]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:44.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:46.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:46 smithi031 ceph-mon[37345]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:46.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:46 smithi026 ceph-mon[29210]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:48.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:48 smithi031 ceph-mon[37345]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:48.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:48 smithi026 ceph-mon[29210]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:49.947 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:49 smithi031 ceph-mon[37345]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:50.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:49 smithi026 ceph-mon[29210]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:50.968 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-03-24T10:38:50.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.967+0000 7fe751ffb700 1 -- 172.21.15.31:0/2010804009 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe7540611f0 con 0x7fe74406cc60 2024-03-24T10:38:50.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 msgr2=0x7fe74406f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 0x7fe74406f120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe740005090 tx=0x7fe740005960 comp rx=0 tx=0).stop 2024-03-24T10:38:50.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 msgr2=0x7fe754074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe74800b480 tx=0x7fe74800b790 comp rx=0 tx=0).stop 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 shutdown_connections 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe74406cc60 0x7fe74406f120 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe754100ae0 0x7fe754074b00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 --2- 172.21.15.31:0/2010804009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe754101ce0 0x7fe7540731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 >> 172.21.15.31:0/2010804009 conn(0x7fe7540fc0a0 msgr2=0x7fe754105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 shutdown_connections 2024-03-24T10:38:50.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:50.970+0000 7fe75aae9700 1 -- 172.21.15.31:0/2010804009 wait complete. 2024-03-24T10:38:51.828 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T10:38:51.830 INFO:tasks.cephadm:Deploying osd.4 on smithi031 with /dev/vg_nvme/lv_3... 2024-03-24T10:38:51.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T10:38:52.078 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:38:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:51 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:51 smithi031 ceph-mon[37345]: from='osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:38:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:51 smithi031 ceph-mon[37345]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:38:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:51 smithi031 ceph-mon[37345]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:52.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:52.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:51 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:52.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:51 smithi026 ceph-mon[29210]: from='osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:38:52.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:51 smithi026 ceph-mon[29210]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:38:52.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:51 smithi026 ceph-mon[29210]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:53.129 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:53.130 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:53.130 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-3[43438]: 2024-03-24T10:38:52.955+0000 7fe23659e700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T10:38:53.130 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 24 10:38:52 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-3[43438]: 2024-03-24T10:38:52.961+0000 7fe232b94700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:53.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:52 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:53 smithi031 ceph-mon[37345]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:38:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:53 smithi031 ceph-mon[37345]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T10:38:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:53 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:53 smithi031 ceph-mon[37345]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:53 smithi026 ceph-mon[29210]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:38:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:53 smithi026 ceph-mon[29210]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T10:38:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:53 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:53 smithi026 ceph-mon[29210]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:38:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T10:38:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T10:38:54.933 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-24T10:38:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-24T10:38:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131396 s, 798 MB/s 2024-03-24T10:38:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] boot 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T10:38:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:54 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:38:55.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:38:55.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:55.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: osd.3 [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] boot 2024-03-24T10:38:55.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T10:38:55.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:54 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:38:55.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-03-24T10:38:56.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:55 smithi031 ceph-mon[37345]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T10:38:56.265 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:55 smithi031 ceph-mon[37345]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:38:56.318 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:38:56.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:55 smithi026 ceph-mon[29210]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T10:38:56.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:55 smithi026 ceph-mon[29210]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:38:57.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T10:38:57.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi031 2024-03-24T10:38:57.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.346 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi031 to 14917M 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:57.347 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:56 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T10:38:57.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi031 2024-03-24T10:38:57.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi031 to 14917M 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:57.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:56 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 -- 172.21.15.31:0/685407125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 msgr2=0x7f82f0101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 --2- 172.21.15.31:0/685407125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f0101040 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009910 tx=0x7f82e0009c20 comp rx=0 tx=0).stop 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 -- 172.21.15.31:0/685407125 shutdown_connections 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 --2- 172.21.15.31:0/685407125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f0101040 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 --2- 172.21.15.31:0/685407125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f00ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 -- 172.21.15.31:0/685407125 >> 172.21.15.31:0/685407125 conn(0x7f82f00faf80 msgr2=0x7f82f00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 -- 172.21.15.31:0/685407125 shutdown_connections 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.845+0000 7f82f87dc700 1 -- 172.21.15.31:0/685407125 wait complete. 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 Processor -- start 2024-03-24T10:38:57.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 -- start start 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f0069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f019b800 con 0x7f82f00ff9e0 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f87dc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f019b940 con 0x7f82f0100be0 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48074/0 (socket says 172.21.15.31:48074) 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 -- 172.21.15.31:0/3789860016 learned_addr learned my addr 172.21.15.31:0/3789860016 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f6578700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f0069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 -- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 msgr2=0x7f82f0069b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f0069b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f5d77700 1 -- 172.21.15.31:0/3789860016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00095f0 con 0x7f82f0100be0 2024-03-24T10:38:57.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.846+0000 7f82f6578700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f0069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82f5d77700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82e0006010 tx=0x7f82e000be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e001d070 con 0x7f82f0100be0 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f019bb70 con 0x7f82f0100be0 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82e000f460 con 0x7f82f0100be0 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e0021640 con 0x7f82f0100be0 2024-03-24T10:38:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.847+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f019c0e0 con 0x7f82f0100be0 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.848+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f018f560 con 0x7f82f0100be0 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.848+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f82e000fab0 con 0x7f82f0100be0 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.849+0000 7f82e77fe700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 0x7f82dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.849+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f82e008cf60 con 0x7f82f0100be0 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.849+0000 7f82f6578700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 0x7f82dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:57.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.849+0000 7f82f6578700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 0x7f82dc06f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f0100a40 tx=0x7f82ec005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:57.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:57.854+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f82e00c09f0 con 0x7f82f0100be0 2024-03-24T10:38:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:38:58.066+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f82f00611f0 con 0x7f82dc06cce0 2024-03-24T10:38:58.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:57 smithi031 ceph-mon[37345]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T10:38:58.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:57 smithi031 ceph-mon[37345]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:38:58.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:57 smithi026 ceph-mon[29210]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T10:38:58.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:57 smithi026 ceph-mon[29210]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: osdmap e31: 4 total, 4 up, 4 in 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: osdmap e31: 4 total, 4 up, 4 in 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:59.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:00.160 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:38:59 smithi031 ceph-mon[37345]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-24T10:39:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:38:59 smithi026 ceph-mon[29210]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-24T10:39:01.386 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:01 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/2426268372' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]: dispatch 2024-03-24T10:39:01.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:01 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]: dispatch 2024-03-24T10:39:01.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:01 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]': finished 2024-03-24T10:39:01.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:01 smithi031 ceph-mon[37345]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T10:39:01.387 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:01 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:01 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2426268372' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]: dispatch 2024-03-24T10:39:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:01 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]: dispatch 2024-03-24T10:39:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:01 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e08bc7e-2981-441e-957a-6810a298651b"}]': finished 2024-03-24T10:39:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:01 smithi026 ceph-mon[29210]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T10:39:01.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:01 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:02 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/3544269132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:39:02.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:02 smithi031 ceph-mon[37345]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:39:02.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:02 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/3544269132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:39:02.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:02 smithi026 ceph-mon[29210]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:39:04.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:03 smithi031 ceph-mon[37345]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:39:04.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:03 smithi026 ceph-mon[29210]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:39:06.440 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:06 smithi031 ceph-mon[37345]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:39:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:06 smithi026 ceph-mon[29210]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:39:08.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:07 smithi031 ceph-mon[37345]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T10:39:08.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:07 smithi026 ceph-mon[29210]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T10:39:08.896 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:08 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T10:39:08.896 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:08 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:08.896 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:08 smithi031 ceph-mon[37345]: Deploying daemon osd.4 on smithi031 2024-03-24T10:39:09.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:08 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T10:39:09.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:08 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:09.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:08 smithi026 ceph-mon[29210]: Deploying daemon osd.4 on smithi031 2024-03-24T10:39:09.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:09 smithi031 ceph-mon[37345]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:10.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:09 smithi026 ceph-mon[29210]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:12.033 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:11 smithi031 ceph-mon[37345]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:12.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:11 smithi026 ceph-mon[29210]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:14.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:14 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:14.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:14 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:14.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:14 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:14 smithi031 ceph-mon[37345]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:14 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:14 smithi026 ceph-mon[29210]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:14.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:14 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:16.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:16 smithi031 ceph-mon[37345]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:16.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:16 smithi026 ceph-mon[29210]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:16.784 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 4 on host 'smithi031' 2024-03-24T10:39:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.783+0000 7f82e77fe700 1 -- 172.21.15.31:0/3789860016 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f82f00611f0 con 0x7f82dc06cce0 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.785+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 msgr2=0x7f82dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.785+0000 7f82f87dc700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 0x7f82dc06f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f0100a40 tx=0x7f82ec005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 msgr2=0x7f82f019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82e0006010 tx=0x7f82e000be20 comp rx=0 tx=0).stop 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 shutdown_connections 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f82dc06cce0 0x7f82dc06f1a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82f00ff9e0 0x7f82f0069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 --2- 172.21.15.31:0/3789860016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82f0100be0 0x7f82f019b230 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 >> 172.21.15.31:0/3789860016 conn(0x7f82f00faf80 msgr2=0x7f82f0104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 shutdown_connections 2024-03-24T10:39:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:16.786+0000 7f82f87dc700 1 -- 172.21.15.31:0/3789860016 wait complete. 2024-03-24T10:39:17.460 DEBUG:teuthology.orchestra.run.smithi031:osd.4> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T10:39:17.463 INFO:tasks.cephadm:Deploying osd.5 on smithi031 with /dev/vg_nvme/lv_2... 2024-03-24T10:39:17.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T10:39:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:39:18.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:39:18.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:39:18.039 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.040 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.040 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:17 smithi031 ceph-mon[37345]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:39:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:39:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:18.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:17 smithi026 ceph-mon[29210]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:19.051 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-mon[37345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T10:39:19.051 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-mon[37345]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T10:39:19.051 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:19.051 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-mon[37345]: from='osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:19.051 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-mon[37345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:19.051 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-4[49906]: 2024-03-24T10:39:18.767+0000 7f662a74c700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T10:39:19.051 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 24 10:39:18 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-4[49906]: 2024-03-24T10:39:18.774+0000 7f6622d3a700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:39:19.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:18 smithi026 ceph-mon[29210]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T10:39:19.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:18 smithi026 ceph-mon[29210]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T10:39:19.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:18 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:19.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:18 smithi026 ceph-mon[29210]: from='osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:19.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:18 smithi026 ceph-mon[29210]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:19.839 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T10:39:19.839 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T10:39:19.839 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-24T10:39:19.839 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-24T10:39:19.839 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125041 s, 839 MB/s 2024-03-24T10:39:19.840 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-24T10:39:20.054 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:19 smithi031 ceph-mon[37345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:39:20.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:19 smithi031 ceph-mon[37345]: osdmap e34: 5 total, 4 up, 5 in 2024-03-24T10:39:20.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:20.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:20.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:19 smithi031 ceph-mon[37345]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:20.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:19 smithi026 ceph-mon[29210]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:39:20.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:19 smithi026 ceph-mon[29210]: osdmap e34: 5 total, 4 up, 5 in 2024-03-24T10:39:20.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:20.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:20.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:19 smithi026 ceph-mon[29210]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T10:39:21.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:20 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:39:21.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:20 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:39:21.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:20 smithi031 ceph-mon[37345]: osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] boot 2024-03-24T10:39:21.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:20 smithi031 ceph-mon[37345]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T10:39:21.055 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:20 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:20 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:39:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:20 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:39:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:20 smithi026 ceph-mon[29210]: osd.4 [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] boot 2024-03-24T10:39:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:20 smithi026 ceph-mon[29210]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T10:39:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:39:21.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-03-24T10:39:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:39:22.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:21 smithi031 ceph-mon[37345]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T10:39:22.037 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:21 smithi031 ceph-mon[37345]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:22.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:21 smithi026 ceph-mon[29210]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T10:39:22.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:21 smithi026 ceph-mon[29210]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi031 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi031 to 7458M 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.399 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:23.400 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:23 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi031 2024-03-24T10:39:23.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi031 to 7458M 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:23.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:23 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:23.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 -- 172.21.15.31:0/1957093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 msgr2=0x7f3678101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 --2- 172.21.15.31:0/1957093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f3678101030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f366c009910 tx=0x7f366c009c20 comp rx=0 tx=0).stop 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 -- 172.21.15.31:0/1957093874 shutdown_connections 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 --2- 172.21.15.31:0/1957093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f3678101030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 --2- 172.21.15.31:0/1957093874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 0x7f36780ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 -- 172.21.15.31:0/1957093874 >> 172.21.15.31:0/1957093874 conn(0x7f36780faf70 msgr2=0x7f36780fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 -- 172.21.15.31:0/1957093874 shutdown_connections 2024-03-24T10:39:23.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.612+0000 7f367e8f8700 1 -- 172.21.15.31:0/1957093874 wait complete. 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 Processor -- start 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 -- start start 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 0x7f3678195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3678195e60 con 0x7f36780ff9d0 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f367e8f8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3678195fa0 con 0x7f3678100bd0 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56586/0 (socket says 172.21.15.31:56586) 2024-03-24T10:39:23.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 -- 172.21.15.31:0/3642269351 learned_addr learned my addr 172.21.15.31:0/3642269351 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f3677fff700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 0x7f3678195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 -- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 msgr2=0x7f3678195390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 0x7f3678195390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.613+0000 7f36777fe700 1 -- 172.21.15.31:0/3642269351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0095f0 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f36777fe700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f366c006010 tx=0x7f366c00be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366c01d070 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367819a9b0 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f366c00f460 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366c021640 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.614+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367819aef0 con 0x7f3678100bd0 2024-03-24T10:39:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.615+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367818f550 con 0x7f3678100bd0 2024-03-24T10:39:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.615+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f366c00fab0 con 0x7f3678100bd0 2024-03-24T10:39:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.615+0000 7f36757fa700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 0x7f366006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.616+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f366c08d110 con 0x7f3678100bd0 2024-03-24T10:39:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.616+0000 7f3677fff700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 0x7f366006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.616+0000 7f3677fff700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 0x7f366006f1a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3678100a30 tx=0x7f3668005d70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.620+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f366c05b460 con 0x7f3678100bd0 2024-03-24T10:39:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:23.833+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f36780611f0 con 0x7f366006cce0 2024-03-24T10:39:24.417 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:24 smithi031 ceph-mon[37345]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:24.417 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:39:24.417 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:39:24.418 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:24 smithi026 ceph-mon[29210]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:39:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:39:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:25.317 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:25 smithi031 ceph-mon[37345]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:25.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:25 smithi026 ceph-mon[29210]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:26.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:26 smithi031 ceph-mon[37345]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:26 smithi026 ceph-mon[29210]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/1427802650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]: dispatch 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]: dispatch 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]': finished 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:27.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:27 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2825636006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/1427802650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]: dispatch 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]: dispatch 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "956ded5d-73fa-44a5-9442-3f02c11d0683"}]': finished 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:27 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/2825636006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:39:28.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:28 smithi031 ceph-mon[37345]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:28.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:28 smithi026 ceph-mon[29210]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:29.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:29.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:30.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:30 smithi031 ceph-mon[37345]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:30.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:30 smithi026 ceph-mon[29210]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:31.823 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:31 smithi031 ceph-mon[37345]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:32.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:31 smithi026 ceph-mon[29210]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:33.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T10:39:33.646 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:33 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T10:39:33.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:33 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:34.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:34 smithi031 ceph-mon[37345]: Deploying daemon osd.5 on smithi031 2024-03-24T10:39:34.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:34 smithi031 ceph-mon[37345]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:34.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:34 smithi026 ceph-mon[29210]: Deploying daemon osd.5 on smithi031 2024-03-24T10:39:34.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:34 smithi026 ceph-mon[29210]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:36.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:36 smithi031 ceph-mon[37345]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:36 smithi026 ceph-mon[29210]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:38.074 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:37 smithi031 ceph-mon[37345]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:38.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:37 smithi026 ceph-mon[29210]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:39.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:39.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:39.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:39 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:39 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:40.886 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:40 smithi031 ceph-mon[37345]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:40 smithi026 ceph-mon[29210]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:42.062 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:41 smithi026 ceph-mon[29210]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:42.069 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:41 smithi031 ceph-mon[37345]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:42.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.395+0000 7f36757fa700 1 -- 172.21.15.31:0/3642269351 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f36780611f0 con 0x7f366006cce0 2024-03-24T10:39:42.396 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 5 on host 'smithi031' 2024-03-24T10:39:42.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 msgr2=0x7f366006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:42.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 0x7f366006f1a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3678100a30 tx=0x7f3668005d70 comp rx=0 tx=0).stop 2024-03-24T10:39:42.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 msgr2=0x7f36781958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:42.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f366c006010 tx=0x7f366c00be20 comp rx=0 tx=0).stop 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 shutdown_connections 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f366006cce0 0x7f366006f1a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36780ff9d0 0x7f3678195390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 --2- 172.21.15.31:0/3642269351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3678100bd0 0x7f36781958d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 >> 172.21.15.31:0/3642269351 conn(0x7f36780faf70 msgr2=0x7f36781041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 shutdown_connections 2024-03-24T10:39:42.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:39:42.397+0000 7f367e8f8700 1 -- 172.21.15.31:0/3642269351 wait complete. 2024-03-24T10:39:43.019 DEBUG:teuthology.orchestra.run.smithi031:osd.5> sudo journalctl -f -n 0 -u ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T10:39:43.021 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-24T10:39:43.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T10:39:43.245 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:39:43.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:39:43.683 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:39:43.683 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.683 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:43.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:44.660 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: from='osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:44.661 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:44 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:44.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T10:39:44.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T10:39:44.688 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:44.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: from='osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:44.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-24T10:39:44.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:44.689 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:44.689 INFO:journalctl@ceph.osd.5.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-5[56492]: 2024-03-24T10:39:44.365+0000 7f8e7cf5f700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T10:39:44.689 INFO:journalctl@ceph.osd.5.smithi031.stdout:Mar 24 10:39:44 smithi031 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-5[56492]: 2024-03-24T10:39:44.373+0000 7f8e77551700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 -- 172.21.15.26:0/3011011700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 msgr2=0x7f2358101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3011011700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358101030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2354009910 tx=0x7f2354009c20 comp rx=0 tx=0).stop 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 -- 172.21.15.26:0/3011011700 shutdown_connections 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3011011700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358101030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3011011700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 0x7f23580ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 -- 172.21.15.26:0/3011011700 >> 172.21.15.26:0/3011011700 conn(0x7f23580faf70 msgr2=0x7f23580fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 -- 172.21.15.26:0/3011011700 shutdown_connections 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.882+0000 7f235f3eb700 1 -- 172.21.15.26:0/3011011700 wait complete. 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 Processor -- start 2024-03-24T10:39:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 -- start start 2024-03-24T10:39:44.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 0x7f2358196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358197060 con 0x7f2358100bd0 2024-03-24T10:39:44.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235f3eb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581971a0 con 0x7f23580ff9d0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36756/0 (socket says 172.21.15.26:36756) 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 -- 172.21.15.26:0/3054317619 learned_addr learned my addr 172.21.15.26:0/3054317619 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235d187700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 0x7f2358196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 -- 172.21.15.26:0/3054317619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 msgr2=0x7f2358196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 0x7f2358196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 -- 172.21.15.26:0/3054317619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23540095f0 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f235c986700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f23540092e0 tx=0x7f235400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.883+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235401d070 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.884+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f235400f460 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.884+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235819bb50 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.884+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2354021640 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.884+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235819c0c0 con 0x7f2358100bd0 2024-03-24T10:39:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.885+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235804ea80 con 0x7f2358100bd0 2024-03-24T10:39:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.885+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f235400fad0 con 0x7f2358100bd0 2024-03-24T10:39:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.885+0000 7f234a7fc700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 0x7f234406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.886+0000 7f235d187700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 0x7f234406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.886+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f235408ceb0 con 0x7f2358100bd0 2024-03-24T10:39:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.886+0000 7f235d187700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 0x7f234406f1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f234c005950 tx=0x7f234c0058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:44.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:44.890+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f235405b290 con 0x7f2358100bd0 2024-03-24T10:39:45.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.101+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f235819c300 con 0x7f2358100bd0 2024-03-24T10:39:45.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.101+0000 7f234a7fc700 1 -- 172.21.15.26:0/3054317619 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f23540260a0 con 0x7f2358100bd0 2024-03-24T10:39:45.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 msgr2=0x7f234406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 0x7f234406f1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f234c005950 tx=0x7f234c0058e0 comp rx=0 tx=0).stop 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 msgr2=0x7f2358196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f23540092e0 tx=0x7f235400be20 comp rx=0 tx=0).stop 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 shutdown_connections 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f234406cce0 0x7f234406f1a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23580ff9d0 0x7f2358196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 --2- 172.21.15.26:0/3054317619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2358100bd0 0x7f2358196a40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 >> 172.21.15.26:0/3054317619 conn(0x7f23580faf70 msgr2=0x7f2358068dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 shutdown_connections 2024-03-24T10:39:45.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:45.103+0000 7f235f3eb700 1 -- 172.21.15.26:0/3054317619 wait complete. 2024-03-24T10:39:45.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:45 smithi031 ceph-mon[37345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:39:45.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:45 smithi031 ceph-mon[37345]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T10:39:45.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:45 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:45.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:45 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:45 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3054317619' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:45 smithi026 ceph-mon[29210]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:45 smithi026 ceph-mon[29210]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:45 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:45 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:45 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3054317619' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:39:45.981 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1711276759,"num_in_osds":6,"osd_in_since":1711276766,"num_remapped_pgs":0} 2024-03-24T10:39:46.536 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: purged_snaps scrub starts 2024-03-24T10:39:46.536 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: purged_snaps scrub ok 2024-03-24T10:39:46.536 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] boot 2024-03-24T10:39:46.537 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T10:39:46.537 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:46.537 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:46 smithi031 ceph-mon[37345]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: purged_snaps scrub starts 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: purged_snaps scrub ok 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: osd.5 [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] boot 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:39:46.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:46 smithi026 ceph-mon[29210]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:39:46.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T10:39:47.216 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:39:47.681 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.682 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:47.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: Detected new or changed devices on smithi031 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: Adjusting osd_memory_target on smithi031 to 4972M 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:48 smithi026 ceph-mon[29210]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: Detected new or changed devices on smithi031 2024-03-24T10:39:48.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: Adjusting osd_memory_target on smithi031 to 4972M 2024-03-24T10:39:48.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:48.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:48.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:39:48.701 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:48 smithi031 ceph-mon[37345]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2938055517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 msgr2=0x7fdbd4100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2938055517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4100ef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009910 tx=0x7fdbc4009c20 comp rx=0 tx=0).stop 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2938055517 shutdown_connections 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2938055517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 0x7fdbd4102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2938055517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4100ef0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2938055517 >> 172.21.15.26:0/2938055517 conn(0x7fdbd40fc090 msgr2=0x7fdbd40fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2938055517 shutdown_connections 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2938055517 wait complete. 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.915+0000 7fdbdacd5700 1 Processor -- start 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 -- start start 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 0x7fdbd40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40736e0 con 0x7fdbd4100ad0 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd4073820 con 0x7fdbd4101cd0 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36782/0 (socket says 172.21.15.26:36782) 2024-03-24T10:39:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 -- 172.21.15.26:0/2658630677 learned_addr learned my addr 172.21.15.26:0/2658630677 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:39:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd3fff700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 0x7fdbd40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 -- 172.21.15.26:0/2658630677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 msgr2=0x7fdbd40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 0x7fdbd40731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 -- 172.21.15.26:0/2658630677 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40095f0 con 0x7fdbd4100ad0 2024-03-24T10:39:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd8a71700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40059d0 tx=0x7fdbc400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc401d070 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbc4005dc0 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd4073aa0 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.916+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc4021410 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.917+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd4073f90 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.918+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd4190650 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.918+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fdbc400f840 con 0x7fdbd4100ad0 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.918+0000 7fdbd1ffb700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 0x7fdbbc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:48.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.918+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdbc408cf60 con 0x7fdbd4100ad0 2024-03-24T10:39:48.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.918+0000 7fdbd3fff700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 0x7fdbbc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:48.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.919+0000 7fdbd3fff700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 0x7fdbbc06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8005fd0 tx=0x7fdbc8005f60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:48.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:48.923+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdbc40c09f0 con 0x7fdbd4100ad0 2024-03-24T10:39:49.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.125+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fdbd404ea80 con 0x7fdbd4100ad0 2024-03-24T10:39:49.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.125+0000 7fdbd1ffb700 1 -- 172.21.15.26:0/2658630677 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7fdbc40260a0 con 0x7fdbd4100ad0 2024-03-24T10:39:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 msgr2=0x7fdbbc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 0x7fdbbc06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8005fd0 tx=0x7fdbc8005f60 comp rx=0 tx=0).stop 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 msgr2=0x7fdbd4074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40059d0 tx=0x7fdbc400be20 comp rx=0 tx=0).stop 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 shutdown_connections 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbbc06cce0 0x7fdbbc06f1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4100ad0 0x7fdbd4074b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 --2- 172.21.15.26:0/2658630677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbd4101cd0 0x7fdbd40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 >> 172.21.15.26:0/2658630677 conn(0x7fdbd40fc090 msgr2=0x7fdbd41052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 shutdown_connections 2024-03-24T10:39:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:49.127+0000 7fdbdacd5700 1 -- 172.21.15.26:0/2658630677 wait complete. 2024-03-24T10:39:49.900 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:49 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2658630677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:39:49.922 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1711276785,"num_in_osds":6,"osd_in_since":1711276766,"num_remapped_pgs":0} 2024-03-24T10:39:49.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:39:49.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:49 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2658630677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:39:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:39:50.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:50 smithi031 ceph-mon[37345]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:50 smithi026 ceph-mon[29210]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 -- 172.21.15.26:0/3505504585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ff320 msgr2=0x7f2b8c0ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/3505504585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c0ff700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c210 tx=0x7f2b7c00c520 comp rx=0 tx=0).stop 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 -- 172.21.15.26:0/3505504585 shutdown_connections 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/3505504585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/3505504585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c0ff700 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 -- 172.21.15.26:0/3505504585 >> 172.21.15.26:0/3505504585 conn(0x7f2b8c075c10 msgr2=0x7f2b8c076020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 -- 172.21.15.26:0/3505504585 shutdown_connections 2024-03-24T10:39:51.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.824+0000 7f2b8b59e700 1 -- 172.21.15.26:0/3505504585 wait complete. 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 Processor -- start 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 -- start start 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c19d0b0 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.825+0000 7f2b8b59e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c196690 con 0x7f2b8c0ff320 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36802/0 (socket says 172.21.15.26:36802) 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 -- 172.21.15.26:0/200645197 learned_addr learned my addr 172.21.15.26:0/200645197 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b8a59c700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 -- 172.21.15.26:0/200645197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ff320 msgr2=0x7f2b8c19c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c19c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:51.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 -- 172.21.15.26:0/200645197 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c00ae10 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b89d9b700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400d900 tx=0x7f2b7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b74009900 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c196970 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b74010460 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b7400f5f0 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.826+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c196ec0 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.827+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c107530 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.828+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f2b740105d0 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.828+0000 7f2b837fe700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 0x7f2b7806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.828+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2b7408ba90 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.828+0000 7f2b8a59c700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 0x7f2b7806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:51.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.829+0000 7f2b8a59c700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 0x7f2b7806f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c016040 tx=0x7f2b7c00c920 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:51.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:51.833+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2b74059c30 con 0x7f2b8c0ffcd0 2024-03-24T10:39:51.833 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:51 smithi026 ceph-mon[29210]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.037+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b8c100670 con 0x7f2b8c0ffcd0 2024-03-24T10:39:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.038+0000 7f2b837fe700 1 -- 172.21.15.26:0/200645197 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11287 (secure 0 0 0) 0x7f2b740597c0 con 0x7f2b8c0ffcd0 2024-03-24T10:39:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:39:52.039 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:33:54.305014+0000","modified":"2024-03-24T10:39:46.367656+0000","last_up_change":"2024-03-24T10:39:45.361543+0000","last_in_change":"2024-03-24T10:39:26.209955+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:38:31.976899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"20a15cb6-4489-4794-a110-50a62c92cc96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6803","nonce":4085968790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6805","nonce":4085968790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6809","nonce":4085968790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6807","nonce":4085968790}]},"public_addr":"172.21.15.26:6803/4085968790","cluster_addr":"172.21.15.26:6805/4085968790","heartbeat_back_addr":"172.21.15.26:6809/4085968790","heartbeat_front_addr":"172.21.15.26:6807/4085968790","state":["exists","up"]},{"osd":1,"uuid":"d1ad8eb2-18e8-4012-93f5-417d61a2832e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6811","nonce":551068105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6813","nonce":551068105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6817","nonce":551068105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6815","nonce":551068105}]},"public_addr":"172.21.15.26:6811/551068105","cluster_addr":"172.21.15.26:6813/551068105","heartbeat_back_addr":"172.21.15.26:6817/551068105","heartbeat_front_addr":"172.21.15.26:6815/551068105","state":["exists","up"]},{"osd":2,"uuid":"c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6819","nonce":291489572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6821","nonce":291489572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6825","nonce":291489572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6823","nonce":291489572}]},"public_addr":"172.21.15.26:6819/291489572","cluster_addr":"172.21.15.26:6821/291489572","heartbeat_back_addr":"172.21.15.26:6825/291489572","heartbeat_front_addr":"172.21.15.26:6823/291489572","state":["exists","up"]},{"osd":3,"uuid":"fd6ae4bc-dd6b-4dbc-ac84-b10495658433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6801","nonce":2495297239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6803","nonce":2495297239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6807","nonce":2495297239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6805","nonce":2495297239}]},"public_addr":"172.21.15.31:6801/2495297239","cluster_addr":"172.21.15.31:6803/2495297239","heartbeat_back_addr":"172.21.15.31:6807/2495297239","heartbeat_front_addr":"172.21.15.31:6805/2495297239","state":["exists","up"]},{"osd":4,"uuid":"0e08bc7e-2981-441e-957a-6810a298651b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6809","nonce":3345716330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6811","nonce":3345716330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6815","nonce":3345716330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6813","nonce":3345716330}]},"public_addr":"172.21.15.31:6809/3345716330","cluster_addr":"172.21.15.31:6811/3345716330","heartbeat_back_addr":"172.21.15.31:6815/3345716330","heartbeat_front_addr":"172.21.15.31:6813/3345716330","state":["exists","up"]},{"osd":5,"uuid":"956ded5d-73fa-44a5-9442-3f02c11d0683","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6817","nonce":2153668083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6819","nonce":2153668083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6823","nonce":2153668083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6821","nonce":2153668083}]},"public_addr":"172.21.15.31:6817/2153668083","cluster_addr":"172.21.15.31:6819/2153668083","heartbeat_back_addr":"172.21.15.31:6823/2153668083","heartbeat_front_addr":"172.21.15.31:6821/2153668083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:32.191494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:01.114679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:28.646628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:52.463191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:18.061331+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:43.503559+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/2057585064":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6801/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3322398799":"2024-03-25T10:36:13.638224+0000","172.21.15.26:0/884630952":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3967355473":"2024-03-25T10:35:03.260951+0000","172.21.15.26:6801/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/105140095":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/2873124826":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/393242092":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/3817743035":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6801/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/787137094":"2024-03-25T10:35:03.260951+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 msgr2=0x7f2b7806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 0x7f2b7806f1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c016040 tx=0x7f2b7c00c920 comp rx=0 tx=0).stop 2024-03-24T10:39:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 msgr2=0x7f2b8c19ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400d900 tx=0x7f2b7400dc10 comp rx=0 tx=0).stop 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 shutdown_connections 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2b7806ccf0 0x7f2b7806f1b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b8c0ff320 0x7f2b8c19c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 --2- 172.21.15.26:0/200645197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b8c0ffcd0 0x7f2b8c19ca90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 >> 172.21.15.26:0/200645197 conn(0x7f2b8c075c10 msgr2=0x7f2b8c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 shutdown_connections 2024-03-24T10:39:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:52.040+0000 7f2b8b59e700 1 -- 172.21.15.26:0/200645197 wait complete. 2024-03-24T10:39:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:51 smithi031 ceph-mon[37345]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:52.782 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:38:31.976899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:39:52.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T10:39:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:39:53.117 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:52 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/200645197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:53.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:52 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/200645197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:54.129 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:53 smithi026 ceph-mon[29210]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:53 smithi031 ceph-mon[37345]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:54.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 -- 172.21.15.26:0/2444546781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 msgr2=0x7ff78810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 --2- 172.21.15.26:0/2444546781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff78810aac0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff778009910 tx=0x7ff778009c20 comp rx=0 tx=0).stop 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 -- 172.21.15.26:0/2444546781 shutdown_connections 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 --2- 172.21.15.26:0/2444546781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff78810aac0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 --2- 172.21.15.26:0/2444546781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 0x7ff788102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 -- 172.21.15.26:0/2444546781 >> 172.21.15.26:0/2444546781 conn(0x7ff788076640 msgr2=0x7ff7880fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 -- 172.21.15.26:0/2444546781 shutdown_connections 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.646+0000 7ff78f570700 1 -- 172.21.15.26:0/2444546781 wait complete. 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 Processor -- start 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 -- start start 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 0x7ff7880ff910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788100390 con 0x7ff7881025d0 2024-03-24T10:39:54.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78f570700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788100500 con 0x7ff788101cb0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58146/0 (socket says 172.21.15.26:58146) 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 -- 172.21.15.26:0/1694577371 learned_addr learned my addr 172.21.15.26:0/1694577371 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78d30c700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 0x7ff7880ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 -- 172.21.15.26:0/1694577371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 msgr2=0x7ff7880ff910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 0x7ff7880ff910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 -- 172.21.15.26:0/1694577371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780095f0 con 0x7ff7881025d0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff78cb0b700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff77800bd70 tx=0x7ff77800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.647+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77801d070 con 0x7ff7881025d0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.648+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff77800f460 con 0x7ff7881025d0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.648+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff778017630 con 0x7ff7881025d0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.648+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788100780 con 0x7ff7881025d0 2024-03-24T10:39:54.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.648+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7881a2f50 con 0x7ff7881025d0 2024-03-24T10:39:54.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.648+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff788068cc0 con 0x7ff7881025d0 2024-03-24T10:39:54.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.649+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ff77800f5d0 con 0x7ff7881025d0 2024-03-24T10:39:54.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.650+0000 7ff77e7fc700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 0x7ff77406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.650+0000 7ff78d30c700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 0x7ff77406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.650+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff77808d130 con 0x7ff7881025d0 2024-03-24T10:39:54.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.650+0000 7ff78d30c700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 0x7ff77406f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff784007da0 tx=0x7ff7840072d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.654+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff7780578c0 con 0x7ff7881025d0 2024-03-24T10:39:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.855+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff788100710 con 0x7ff7881025d0 2024-03-24T10:39:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.855+0000 7ff77e7fc700 1 -- 172.21.15.26:0/1694577371 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff778005740 con 0x7ff7881025d0 2024-03-24T10:39:54.856 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 msgr2=0x7ff77406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 0x7ff77406f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff784007da0 tx=0x7ff7840072d0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 msgr2=0x7ff7880ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff77800bd70 tx=0x7ff77800be50 comp rx=0 tx=0).stop 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 shutdown_connections 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff77406cce0 0x7ff77406f1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff788101cb0 0x7ff7880ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 --2- 172.21.15.26:0/1694577371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7881025d0 0x7ff7880ffe50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 >> 172.21.15.26:0/1694577371 conn(0x7ff788076640 msgr2=0x7ff788103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 shutdown_connections 2024-03-24T10:39:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:54.857+0000 7ff78f570700 1 -- 172.21.15.26:0/1694577371 wait complete. 2024-03-24T10:39:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:54 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1694577371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:54 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1694577371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.572 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T10:39:55.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T10:39:55.787 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:39:56.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:56 smithi026 ceph-mon[29210]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:56.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:56 smithi031 ceph-mon[37345]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 -- 172.21.15.26:0/1459942469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 msgr2=0x7face40ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 --2- 172.21.15.26:0/1459942469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face40ff200 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7facd4009910 tx=0x7facd4009c20 comp rx=0 tx=0).stop 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 -- 172.21.15.26:0/1459942469 shutdown_connections 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 --2- 172.21.15.26:0/1459942469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 0x7face41022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 --2- 172.21.15.26:0/1459942469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face40ff200 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 -- 172.21.15.26:0/1459942469 >> 172.21.15.26:0/1459942469 conn(0x7face40fa700 msgr2=0x7face40fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 -- 172.21.15.26:0/1459942469 shutdown_connections 2024-03-24T10:39:57.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.433+0000 7face9314700 1 -- 172.21.15.26:0/1459942469 wait complete. 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 Processor -- start 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 -- start start 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 0x7face4196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41971f0 con 0x7face40fede0 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.434+0000 7face9314700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4197330 con 0x7face4101e60 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58168/0 (socket says 172.21.15.26:58168) 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 -- 172.21.15.26:0/351643298 learned_addr learned my addr 172.21.15.26:0/351643298 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:39:57.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face27fc700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 0x7face4196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 -- 172.21.15.26:0/351643298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 msgr2=0x7face4196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 0x7face4196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 -- 172.21.15.26:0/351643298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40095f0 con 0x7face40fede0 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7face2ffd700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7facd4005850 tx=0x7facd400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd401d070 con 0x7face40fede0 2024-03-24T10:39:57.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd4005dc0 con 0x7face40fede0 2024-03-24T10:39:57.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.435+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd4021410 con 0x7face40fede0 2024-03-24T10:39:57.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.436+0000 7face9314700 1 -- 172.21.15.26:0/351643298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face419bd30 con 0x7face40fede0 2024-03-24T10:39:57.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.436+0000 7face9314700 1 -- 172.21.15.26:0/351643298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face419c220 con 0x7face40fede0 2024-03-24T10:39:57.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.437+0000 7face9314700 1 -- 172.21.15.26:0/351643298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4190700 con 0x7face40fede0 2024-03-24T10:39:57.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.438+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7facd400f7b0 con 0x7face40fede0 2024-03-24T10:39:57.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.438+0000 7faccbfff700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 0x7faccc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.438+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7facd408cc40 con 0x7face40fede0 2024-03-24T10:39:57.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.438+0000 7face27fc700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 0x7faccc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.439+0000 7face27fc700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 0x7faccc06f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7facd8009a10 tx=0x7facd8009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.443+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7facd405ae60 con 0x7face40fede0 2024-03-24T10:39:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.723+0000 7face9314700 1 -- 172.21.15.26:0/351643298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7face404ea80 con 0x7face40fede0 2024-03-24T10:39:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.728+0000 7faccbfff700 1 -- 172.21.15.26:0/351643298 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7facd405a9f0 con 0x7face40fede0 2024-03-24T10:39:57.728 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-03-24T10:39:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQD9AgBmQYU6KxAAxfdgE4hr0TnvM3xh2MAVHQ== 2024-03-24T10:39:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 msgr2=0x7faccc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 0x7faccc06f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7facd8009a10 tx=0x7facd8009190 comp rx=0 tx=0).stop 2024-03-24T10:39:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 msgr2=0x7face4196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7facd4005850 tx=0x7facd400bd70 comp rx=0 tx=0).stop 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 shutdown_connections 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7faccc06cce0 0x7faccc06f1a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7face40fede0 0x7face4196690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 --2- 172.21.15.26:0/351643298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7face4101e60 0x7face4196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 >> 172.21.15.26:0/351643298 conn(0x7face40fa700 msgr2=0x7face41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 shutdown_connections 2024-03-24T10:39:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:39:57.730+0000 7face9314700 1 -- 172.21.15.26:0/351643298 wait complete. 2024-03-24T10:39:57.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:57 smithi031 ceph-mon[37345]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:57.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:57 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/351643298' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:39:57.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:57 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/351643298' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:39:58.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:57 smithi026 ceph-mon[29210]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:39:58.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:57 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/351643298' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:39:58.002 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:57 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/351643298' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:39:58.376 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:39:58.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T10:39:58.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T10:39:58.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T10:39:58.642 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi031/config 2024-03-24T10:39:59.088 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:59.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:39:59.992 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:39:59 smithi031 ceph-mon[37345]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:00.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:39:59 smithi026 ceph-mon[29210]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:00.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 -- 172.21.15.31:0/3001971868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c0feda0 msgr2=0x7ffb5c0ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 --2- 172.21.15.31:0/3001971868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c0ff1c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009910 tx=0x7ffb4c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:00.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 -- 172.21.15.31:0/3001971868 shutdown_connections 2024-03-24T10:40:00.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 --2- 172.21.15.31:0/3001971868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c101d80 0x7ffb5c1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 --2- 172.21.15.31:0/3001971868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c0ff1c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 -- 172.21.15.31:0/3001971868 >> 172.21.15.31:0/3001971868 conn(0x7ffb5c0fa700 msgr2=0x7ffb5c0fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 -- 172.21.15.31:0/3001971868 shutdown_connections 2024-03-24T10:40:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.226+0000 7ffb616dc700 1 -- 172.21.15.31:0/3001971868 wait complete. 2024-03-24T10:40:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 Processor -- start 2024-03-24T10:40:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 -- start start 2024-03-24T10:40:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c100270 con 0x7ffb5c0feda0 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb616dc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c1003b0 con 0x7ffb5c101d80 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.227+0000 7ffb5a7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33326/0 (socket says 172.21.15.31:33326) 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 -- 172.21.15.31:0/2162373932 learned_addr learned my addr 172.21.15.31:0/2162373932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5affd700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 -- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c0feda0 msgr2=0x7ffb5c101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 -- 172.21.15.31:0/2162373932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0095f0 con 0x7ffb5c101d80 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb5a7fc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c8a0 tx=0x7ffb5000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb5000cea0 con 0x7ffb5c101d80 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb50004830 con 0x7ffb5c101d80 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c100690 con 0x7ffb5c101d80 2024-03-24T10:40:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb500055b0 con 0x7ffb5c101d80 2024-03-24T10:40:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.228+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c19e550 con 0x7ffb5c101d80 2024-03-24T10:40:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.229+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c190660 con 0x7ffb5c101d80 2024-03-24T10:40:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.230+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ffb500049a0 con 0x7ffb5c101d80 2024-03-24T10:40:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.230+0000 7ffb43fff700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 0x7ffb4406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.230+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffb5008bbb0 con 0x7ffb5c101d80 2024-03-24T10:40:00.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.230+0000 7ffb5affd700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 0x7ffb4406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.231+0000 7ffb5affd700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 0x7ffb4406f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c00b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.235+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb50092080 con 0x7ffb5c101d80 2024-03-24T10:40:00.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.526+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ffb5c02cf60 con 0x7ffb5c101d80 2024-03-24T10:40:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.530+0000 7ffb43fff700 1 -- 172.21.15.31:0/2162373932 <== mon.1 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7ffb50059d50 con 0x7ffb5c101d80 2024-03-24T10:40:00.530 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-03-24T10:40:00.531 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAAAwBmLU91HxAA7Kw+tKQZLTNghStL1YYqvw== 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 msgr2=0x7ffb4406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 0x7ffb4406f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c00b580 comp rx=0 tx=0).stop 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 msgr2=0x7ffb5c0ffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c8a0 tx=0x7ffb5000cc60 comp rx=0 tx=0).stop 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 shutdown_connections 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffb4406cce0 0x7ffb4406f1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb5c0feda0 0x7ffb5c101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 --2- 172.21.15.31:0/2162373932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb5c101d80 0x7ffb5c0ffd30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.532+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 >> 172.21.15.31:0/2162373932 conn(0x7ffb5c0fa700 msgr2=0x7ffb5c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.533+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 shutdown_connections 2024-03-24T10:40:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:40:00.533+0000 7ffb616dc700 1 -- 172.21.15.31:0/2162373932 wait complete. 2024-03-24T10:40:00.854 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:00 smithi031 ceph-mon[37345]: overall HEALTH_OK 2024-03-24T10:40:00.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:00 smithi031 ceph-mon[37345]: from='client.? 172.21.15.31:0/2162373932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:40:00.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:00 smithi031 ceph-mon[37345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:40:00.855 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:00 smithi031 ceph-mon[37345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:40:01.145 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:40:01.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T10:40:01.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T10:40:01.190 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T10:40:01.190 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T10:40:01.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T10:40:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:00 smithi026 ceph-mon[29210]: overall HEALTH_OK 2024-03-24T10:40:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:00 smithi026 ceph-mon[29210]: from='client.? 172.21.15.31:0/2162373932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:40:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:00 smithi026 ceph-mon[29210]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:40:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:00 smithi026 ceph-mon[29210]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:40:01.412 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:02.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:01 smithi031 ceph-mon[37345]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:01 smithi026 ceph-mon[29210]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:02.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 -- 172.21.15.26:0/1272695242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100d00 msgr2=0x7f2608104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:02.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 --2- 172.21.15.26:0/1272695242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100d00 0x7f2608104830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009910 tx=0x7f25fc009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:02.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 -- 172.21.15.26:0/1272695242 shutdown_connections 2024-03-24T10:40:02.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 --2- 172.21.15.26:0/1272695242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100d00 0x7f2608104830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:02.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 --2- 172.21.15.26:0/1272695242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100350 0x7f2608100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 -- 172.21.15.26:0/1272695242 >> 172.21.15.26:0/1272695242 conn(0x7f2608075e10 msgr2=0x7f26080fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 -- 172.21.15.26:0/1272695242 shutdown_connections 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.922+0000 7f26106a7700 1 -- 172.21.15.26:0/1272695242 wait complete. 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 Processor -- start 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 -- start start 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100d00 0x7f2608197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608198540 con 0x7f2608100350 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.923+0000 7f26106a7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608198680 con 0x7f2608100d00 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48286/0 (socket says 172.21.15.26:48286) 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 -- 172.21.15.26:0/1260675743 learned_addr learned my addr 172.21.15.26:0/1260675743 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260dc42700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100d00 0x7f2608197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 -- 172.21.15.26:0/1260675743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100d00 msgr2=0x7f2608197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100d00 0x7f2608197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 -- 172.21.15.26:0/1260675743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0095f0 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f260e443700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25f800ada0 tx=0x7f25f800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f800f070 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25f800ae90 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26081a0aa0 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f8005690 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.924+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26081a0eb0 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.925+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2608108230 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.926+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f25f8005900 con 0x7f2608100350 2024-03-24T10:40:02.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.926+0000 7f26077fe700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 0x7f25f006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:02.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.926+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f25f808b8a0 con 0x7f2608100350 2024-03-24T10:40:02.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.926+0000 7f260dc42700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 0x7f25f006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:02.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.926+0000 7f260dc42700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 0x7f25f006f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26080ff7d0 tx=0x7f25fc01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:02.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:02.931+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f25f8055fb0 con 0x7f2608100350 2024-03-24T10:40:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.191+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f26081013a0 con 0x7f2608100350 2024-03-24T10:40:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.194+0000 7f26077fe700 1 -- 172.21.15.26:0/1260675743 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+174002 (secure 0 0 0) 0x7f25f80595d0 con 0x7f2608100350 2024-03-24T10:40:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 msgr2=0x7f25f006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 0x7f25f006f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26080ff7d0 tx=0x7f25fc01a040 comp rx=0 tx=0).stop 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 msgr2=0x7f2608197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25f800ada0 tx=0x7f25f800c330 comp rx=0 tx=0).stop 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 shutdown_connections 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f25f006cce0 0x7f25f006f1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2608100350 0x7f2608197a30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 --2- 172.21.15.26:0/1260675743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2608100d00 0x7f2608197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 >> 172.21.15.26:0/1260675743 conn(0x7f2608075e10 msgr2=0x7f26080fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 shutdown_connections 2024-03-24T10:40:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:03.198+0000 7f26106a7700 1 -- 172.21.15.26:0/1260675743 wait complete. 2024-03-24T10:40:03.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:03 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1260675743' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T10:40:03.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:03 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1260675743' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T10:40:03.853 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi026.bbjcru","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":3570278484},{"type":"v1","addr":"172.21.15.26:6801","nonce":3570278484}]},"active_addr":"172.21.15.26:6801/3570278484","active_change":"2024-03-24T10:36:13.638374+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14238,"name":"smithi031.ucwncv","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.26:8443/","prometheus":"http://172.21.15.26:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1801024643}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2459119745}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":556089098}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2650874935}]}]} 2024-03-24T10:40:03.857 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T10:40:03.857 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T10:40:03.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:40:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:04.551 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:04 smithi026 ceph-mon[29210]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:04.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:04 smithi031 ceph-mon[37345]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:05.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 -- 172.21.15.26:0/871227752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 msgr2=0x7fe69c0ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:05.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 --2- 172.21.15.26:0/871227752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c0ff350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe684009910 tx=0x7fe684009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:05.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 -- 172.21.15.26:0/871227752 shutdown_connections 2024-03-24T10:40:05.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 --2- 172.21.15.26:0/871227752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c0ff350 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:05.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 --2- 172.21.15.26:0/871227752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 0x7fe69c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:05.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 -- 172.21.15.26:0/871227752 >> 172.21.15.26:0/871227752 conn(0x7fe69c075e10 msgr2=0x7fe69c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:05.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 -- 172.21.15.26:0/871227752 shutdown_connections 2024-03-24T10:40:05.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.827+0000 7fe6a1818700 1 -- 172.21.15.26:0/871227752 wait complete. 2024-03-24T10:40:05.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 Processor -- start 2024-03-24T10:40:05.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 -- start start 2024-03-24T10:40:05.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:05.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 0x7fe69c196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:05.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c1975f0 con 0x7fe69c0feed0 2024-03-24T10:40:05.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe6a1818700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c19b340 con 0x7fe69c105ab0 2024-03-24T10:40:05.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48300/0 (socket says 172.21.15.26:48300) 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69affd700 1 -- 172.21.15.26:0/2122569698 learned_addr learned my addr 172.21.15.26:0/2122569698 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69a7fc700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 0x7fe69c196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69affd700 1 -- 172.21.15.26:0/2122569698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 msgr2=0x7fe69c196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.828+0000 7fe69affd700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 0x7fe69c196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe69affd700 1 -- 172.21.15.26:0/2122569698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840095f0 con 0x7fe69c0feed0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe69affd700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00ada0 tx=0x7fe68c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe68c00f070 con 0x7fe69c0feed0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe68c00ae90 con 0x7fe69c0feed0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c19b620 con 0x7fe69c0feed0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe68c0056d0 con 0x7fe69c0feed0 2024-03-24T10:40:05.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.829+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c19bb70 con 0x7fe69c0feed0 2024-03-24T10:40:05.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.831+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c1090a0 con 0x7fe69c0feed0 2024-03-24T10:40:05.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.831+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fe68c005830 con 0x7fe69c0feed0 2024-03-24T10:40:05.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.831+0000 7fe6a0816700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 0x7fe68806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:05.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.831+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe68c08b7e0 con 0x7fe69c0feed0 2024-03-24T10:40:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.831+0000 7fe69a7fc700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 0x7fe68806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.832+0000 7fe69a7fc700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 0x7fe68806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe69c197f80 tx=0x7fe68401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:05.836+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe68c059980 con 0x7fe69c0feed0 2024-03-24T10:40:06.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.040+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe69c1021f0 con 0x7fe69c0feed0 2024-03-24T10:40:06.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.041+0000 7fe6a0816700 1 -- 172.21.15.26:0/2122569698 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11287 (secure 0 0 0) 0x7fe68c059510 con 0x7fe69c0feed0 2024-03-24T10:40:06.041 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:06.042 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:33:54.305014+0000","modified":"2024-03-24T10:39:46.367656+0000","last_up_change":"2024-03-24T10:39:45.361543+0000","last_in_change":"2024-03-24T10:39:26.209955+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:38:31.976899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"20a15cb6-4489-4794-a110-50a62c92cc96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6803","nonce":4085968790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6805","nonce":4085968790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6809","nonce":4085968790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6807","nonce":4085968790}]},"public_addr":"172.21.15.26:6803/4085968790","cluster_addr":"172.21.15.26:6805/4085968790","heartbeat_back_addr":"172.21.15.26:6809/4085968790","heartbeat_front_addr":"172.21.15.26:6807/4085968790","state":["exists","up"]},{"osd":1,"uuid":"d1ad8eb2-18e8-4012-93f5-417d61a2832e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6811","nonce":551068105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6813","nonce":551068105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6817","nonce":551068105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6815","nonce":551068105}]},"public_addr":"172.21.15.26:6811/551068105","cluster_addr":"172.21.15.26:6813/551068105","heartbeat_back_addr":"172.21.15.26:6817/551068105","heartbeat_front_addr":"172.21.15.26:6815/551068105","state":["exists","up"]},{"osd":2,"uuid":"c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6819","nonce":291489572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6821","nonce":291489572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6825","nonce":291489572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6823","nonce":291489572}]},"public_addr":"172.21.15.26:6819/291489572","cluster_addr":"172.21.15.26:6821/291489572","heartbeat_back_addr":"172.21.15.26:6825/291489572","heartbeat_front_addr":"172.21.15.26:6823/291489572","state":["exists","up"]},{"osd":3,"uuid":"fd6ae4bc-dd6b-4dbc-ac84-b10495658433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6801","nonce":2495297239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6803","nonce":2495297239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6807","nonce":2495297239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6805","nonce":2495297239}]},"public_addr":"172.21.15.31:6801/2495297239","cluster_addr":"172.21.15.31:6803/2495297239","heartbeat_back_addr":"172.21.15.31:6807/2495297239","heartbeat_front_addr":"172.21.15.31:6805/2495297239","state":["exists","up"]},{"osd":4,"uuid":"0e08bc7e-2981-441e-957a-6810a298651b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6809","nonce":3345716330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6811","nonce":3345716330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6815","nonce":3345716330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6813","nonce":3345716330}]},"public_addr":"172.21.15.31:6809/3345716330","cluster_addr":"172.21.15.31:6811/3345716330","heartbeat_back_addr":"172.21.15.31:6815/3345716330","heartbeat_front_addr":"172.21.15.31:6813/3345716330","state":["exists","up"]},{"osd":5,"uuid":"956ded5d-73fa-44a5-9442-3f02c11d0683","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6817","nonce":2153668083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6819","nonce":2153668083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6823","nonce":2153668083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6821","nonce":2153668083}]},"public_addr":"172.21.15.31:6817/2153668083","cluster_addr":"172.21.15.31:6819/2153668083","heartbeat_back_addr":"172.21.15.31:6823/2153668083","heartbeat_front_addr":"172.21.15.31:6821/2153668083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:32.191494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:01.114679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:28.646628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:52.463191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:18.061331+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:43.503559+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/2057585064":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6801/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3322398799":"2024-03-25T10:36:13.638224+0000","172.21.15.26:0/884630952":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3967355473":"2024-03-25T10:35:03.260951+0000","172.21.15.26:6801/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/105140095":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/2873124826":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/393242092":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/3817743035":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6801/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/787137094":"2024-03-25T10:35:03.260951+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 msgr2=0x7fe68806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 0x7fe68806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe69c197f80 tx=0x7fe68401a040 comp rx=0 tx=0).stop 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 msgr2=0x7fe69c196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00ada0 tx=0x7fe68c00c330 comp rx=0 tx=0).stop 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 shutdown_connections 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe68806cce0 0x7fe68806f1a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe69c0feed0 0x7fe69c196a20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 --2- 172.21.15.26:0/2122569698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c105ab0 0x7fe69c196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 >> 172.21.15.26:0/2122569698 conn(0x7fe69c075e10 msgr2=0x7fe69c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 shutdown_connections 2024-03-24T10:40:06.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:06.043+0000 7fe6a1818700 1 -- 172.21.15.26:0/2122569698 wait complete. 2024-03-24T10:40:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:06 smithi031 ceph-mon[37345]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:06.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:06 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2122569698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:06 smithi026 ceph-mon[29210]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:06.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:06 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2122569698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.721 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T10:40:06.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:40:06.936 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:08.044 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:07 smithi026 ceph-mon[29210]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:07 smithi031 ceph-mon[37345]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 -- 172.21.15.26:0/3092057361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 msgr2=0x7ffb10106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 --2- 172.21.15.26:0/3092057361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb10106e70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009910 tx=0x7ffb0c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 -- 172.21.15.26:0/3092057361 shutdown_connections 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 --2- 172.21.15.26:0/3092057361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 0x7ffb10100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 --2- 172.21.15.26:0/3092057361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb10106e70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 -- 172.21.15.26:0/3092057361 >> 172.21.15.26:0/3092057361 conn(0x7ffb100fc5a0 msgr2=0x7ffb100fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 -- 172.21.15.26:0/3092057361 shutdown_connections 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 -- 172.21.15.26:0/3092057361 wait complete. 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.576+0000 7ffb186ef700 1 Processor -- start 2024-03-24T10:40:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb186ef700 1 -- start start 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb186ef700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 0x7ffb1019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb186ef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb186ef700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1019d290 con 0x7ffb10106a90 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb186ef700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb10196850 con 0x7ffb10100a90 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48308/0 (socket says 172.21.15.26:48308) 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 -- 172.21.15.26:0/889582257 learned_addr learned my addr 172.21.15.26:0/889582257 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb1648b700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 0x7ffb1019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 -- 172.21.15.26:0/889582257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 msgr2=0x7ffb1019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 0x7ffb1019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 -- 172.21.15.26:0/889582257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0095f0 con 0x7ffb10106a90 2024-03-24T10:40:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.577+0000 7ffb15c8a700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d900 tx=0x7ffb0400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.578+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb040041f0 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.578+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb04004cf0 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.578+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10196b30 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.578+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0400b750 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.578+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb10197080 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.579+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10109090 con 0x7ffb10106a90 2024-03-24T10:40:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.580+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ffb04004750 con 0x7ffb10106a90 2024-03-24T10:40:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.580+0000 7ffb037fe700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 0x7ffafc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.580+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffb0408bde0 con 0x7ffb10106a90 2024-03-24T10:40:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.580+0000 7ffb1648b700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 0x7ffafc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.581+0000 7ffb1648b700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 0x7ffafc06f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006010 tx=0x7ffb0c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.585+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb04059f80 con 0x7ffb10106a90 2024-03-24T10:40:08.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.790+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb101021d0 con 0x7ffb10106a90 2024-03-24T10:40:08.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.791+0000 7ffb037fe700 1 -- 172.21.15.26:0/889582257 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11287 (secure 0 0 0) 0x7ffb04059b10 con 0x7ffb10106a90 2024-03-24T10:40:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:08.792 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:33:54.305014+0000","modified":"2024-03-24T10:39:46.367656+0000","last_up_change":"2024-03-24T10:39:45.361543+0000","last_in_change":"2024-03-24T10:39:26.209955+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:38:31.976899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"20a15cb6-4489-4794-a110-50a62c92cc96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6803","nonce":4085968790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6805","nonce":4085968790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6809","nonce":4085968790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6807","nonce":4085968790}]},"public_addr":"172.21.15.26:6803/4085968790","cluster_addr":"172.21.15.26:6805/4085968790","heartbeat_back_addr":"172.21.15.26:6809/4085968790","heartbeat_front_addr":"172.21.15.26:6807/4085968790","state":["exists","up"]},{"osd":1,"uuid":"d1ad8eb2-18e8-4012-93f5-417d61a2832e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6811","nonce":551068105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6813","nonce":551068105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6817","nonce":551068105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6815","nonce":551068105}]},"public_addr":"172.21.15.26:6811/551068105","cluster_addr":"172.21.15.26:6813/551068105","heartbeat_back_addr":"172.21.15.26:6817/551068105","heartbeat_front_addr":"172.21.15.26:6815/551068105","state":["exists","up"]},{"osd":2,"uuid":"c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6819","nonce":291489572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6821","nonce":291489572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6825","nonce":291489572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6823","nonce":291489572}]},"public_addr":"172.21.15.26:6819/291489572","cluster_addr":"172.21.15.26:6821/291489572","heartbeat_back_addr":"172.21.15.26:6825/291489572","heartbeat_front_addr":"172.21.15.26:6823/291489572","state":["exists","up"]},{"osd":3,"uuid":"fd6ae4bc-dd6b-4dbc-ac84-b10495658433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6801","nonce":2495297239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6803","nonce":2495297239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6807","nonce":2495297239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6805","nonce":2495297239}]},"public_addr":"172.21.15.31:6801/2495297239","cluster_addr":"172.21.15.31:6803/2495297239","heartbeat_back_addr":"172.21.15.31:6807/2495297239","heartbeat_front_addr":"172.21.15.31:6805/2495297239","state":["exists","up"]},{"osd":4,"uuid":"0e08bc7e-2981-441e-957a-6810a298651b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6809","nonce":3345716330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6811","nonce":3345716330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6815","nonce":3345716330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6813","nonce":3345716330}]},"public_addr":"172.21.15.31:6809/3345716330","cluster_addr":"172.21.15.31:6811/3345716330","heartbeat_back_addr":"172.21.15.31:6815/3345716330","heartbeat_front_addr":"172.21.15.31:6813/3345716330","state":["exists","up"]},{"osd":5,"uuid":"956ded5d-73fa-44a5-9442-3f02c11d0683","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6817","nonce":2153668083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6819","nonce":2153668083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6823","nonce":2153668083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6821","nonce":2153668083}]},"public_addr":"172.21.15.31:6817/2153668083","cluster_addr":"172.21.15.31:6819/2153668083","heartbeat_back_addr":"172.21.15.31:6823/2153668083","heartbeat_front_addr":"172.21.15.31:6821/2153668083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:32.191494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:01.114679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:28.646628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:52.463191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:18.061331+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:43.503559+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/2057585064":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6801/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3322398799":"2024-03-25T10:36:13.638224+0000","172.21.15.26:0/884630952":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3967355473":"2024-03-25T10:35:03.260951+0000","172.21.15.26:6801/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/105140095":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/2873124826":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/393242092":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6800/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/3817743035":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6801/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/787137094":"2024-03-25T10:35:03.260951+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 msgr2=0x7ffafc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 0x7ffafc06f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006010 tx=0x7ffb0c01a040 comp rx=0 tx=0).stop 2024-03-24T10:40:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 msgr2=0x7ffb1019cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d900 tx=0x7ffb0400dc10 comp rx=0 tx=0).stop 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 shutdown_connections 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffafc06cce0 0x7ffafc06f1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb10100a90 0x7ffb1019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 --2- 172.21.15.26:0/889582257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffb10106a90 0x7ffb1019cc00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 >> 172.21.15.26:0/889582257 conn(0x7ffb100fc5a0 msgr2=0x7ffb100fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 shutdown_connections 2024-03-24T10:40:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:08.793+0000 7ffb186ef700 1 -- 172.21.15.26:0/889582257 wait complete. 2024-03-24T10:40:09.110 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:08 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/889582257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:09.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:08 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/889582257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:09.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T10:40:09.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T10:40:09.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T10:40:09.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T10:40:09.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T10:40:09.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T10:40:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:09.847 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:09 smithi026 ceph-mon[29210]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:10.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:09 smithi031 ceph-mon[37345]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:11.838 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:11.838 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:12.040 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:11 smithi026 ceph-mon[29210]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:12.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:11 smithi031 ceph-mon[37345]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:13 smithi026 ceph-mon[29210]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:13.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:13 smithi031 ceph-mon[37345]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.130+0000 7f2c9c75c700 1 -- 172.21.15.26:0/2494770606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 msgr2=0x7f2c94106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.130+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/2494770606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c94106e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009910 tx=0x7f2c90009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- 172.21.15.26:0/2494770606 shutdown_connections 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/2494770606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 0x7f2c94100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/2494770606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c94106e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- 172.21.15.26:0/2494770606 >> 172.21.15.26:0/2494770606 conn(0x7f2c940fc5a0 msgr2=0x7f2c940fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- 172.21.15.26:0/2494770606 shutdown_connections 2024-03-24T10:40:15.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- 172.21.15.26:0/2494770606 wait complete. 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 Processor -- start 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- start start 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 0x7f2c940730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94073b30 con 0x7f2c94106a70 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c9c75c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94073c70 con 0x7f2c94100a70 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.131+0000 7f2c99cf7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54374/0 (socket says 172.21.15.26:54374) 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 -- 172.21.15.26:0/606690617 learned_addr learned my addr 172.21.15.26:0/606690617 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c9a4f8700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 0x7f2c940730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 -- 172.21.15.26:0/606690617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 msgr2=0x7f2c940730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 0x7f2c940730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c900095f0 con 0x7f2c94106a70 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c99cf7700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400d8d0 tx=0x7f2c8400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c840098a0 con 0x7f2c94106a70 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c84010460 con 0x7f2c94106a70 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c941a2b70 con 0x7f2c94106a70 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8400f5f0 con 0x7f2c94106a70 2024-03-24T10:40:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.132+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c941a2ed0 con 0x7f2c94106a70 2024-03-24T10:40:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.133+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c9410df70 con 0x7f2c94106a70 2024-03-24T10:40:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f2c84009a00 con 0x7f2c94106a70 2024-03-24T10:40:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 0x7f2c8006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2c8408c320 con 0x7f2c94106a70 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c9a4f8700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 0x7f2c8006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 0x7f2c80074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c80075490 con 0x7f2c800729c0 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c8408c780 con 0x7f2c94106a70 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.134+0000 7f2c9acf9700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 0x7f2c80074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.135+0000 7f2c9a4f8700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 0x7f2c8006f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000c00 tx=0x7f2c9001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.135+0000 7f2c9acf9700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 0x7f2c80074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.135+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== osd.0 v2:172.21.15.26:6802/4085968790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c80075490 con 0x7f2c800729c0 2024-03-24T10:40:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.153+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 --> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c94061470 con 0x7f2c800729c0 2024-03-24T10:40:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.153+0000 7f2c8b7fe700 1 -- 172.21.15.26:0/606690617 <== osd.0 v2:172.21.15.26:6802/4085968790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2c94061470 con 0x7f2c800729c0 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 msgr2=0x7f2c80074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 0x7f2c80074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 msgr2=0x7f2c8006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 0x7f2c8006f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000c00 tx=0x7f2c9001a040 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 msgr2=0x7f2c940735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400d8d0 tx=0x7f2c8400dbe0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 shutdown_connections 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6802/4085968790,v1:172.21.15.26:6803/4085968790] conn(0x7f2c800729c0 0x7f2c80074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2c8006cce0 0x7f2c8006f1a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c94100a70 0x7f2c940730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 --2- 172.21.15.26:0/606690617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94106a70 0x7f2c940735f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 >> 172.21.15.26:0/606690617 conn(0x7f2c940fc5a0 msgr2=0x7f2c940fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 shutdown_connections 2024-03-24T10:40:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.154+0000 7f2c9c75c700 1 -- 172.21.15.26:0/606690617 wait complete. 2024-03-24T10:40:15.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 -- 172.21.15.26:0/1949412985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c105990 msgr2=0x7f826c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 --2- 172.21.15.26:0/1949412985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c105990 0x7f826c105d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f825c009910 tx=0x7f825c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 -- 172.21.15.26:0/1949412985 shutdown_connections 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 --2- 172.21.15.26:0/1949412985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c0ff990 0x7f826c0ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 --2- 172.21.15.26:0/1949412985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c105990 0x7f826c105d70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 -- 172.21.15.26:0/1949412985 >> 172.21.15.26:0/1949412985 conn(0x7f826c0fb4c0 msgr2=0x7f826c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 -- 172.21.15.26:0/1949412985 shutdown_connections 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.682+0000 7f8273d17700 1 -- 172.21.15.26:0/1949412985 wait complete. 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 Processor -- start 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 -- start start 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c105990 0x7f826c0fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0fc7b0 con 0x7f826c0ff990 2024-03-24T10:40:15.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8273d17700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0fc8f0 con 0x7f826c105990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8271ab3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8271ab3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54384/0 (socket says 172.21.15.26:54384) 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.683+0000 7f8271ab3700 1 -- 172.21.15.26:0/2910464217 learned_addr learned my addr 172.21.15.26:0/2910464217 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f82712b2700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c105990 0x7f826c0fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8271ab3700 1 -- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c105990 msgr2=0x7f826c0fc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8271ab3700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c105990 0x7f826c0fc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8271ab3700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0095f0 con 0x7f826c0ff990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8271ab3700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f825c00bd70 tx=0x7f825c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f825c01d070 con 0x7f826c0ff990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f825c00f460 con 0x7f826c0ff990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c0fcb70 con 0x7f826c0ff990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f825c017630 con 0x7f826c0ff990 2024-03-24T10:40:15.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.684+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c1a2ed0 con 0x7f826c0ff990 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.685+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f826c10cf10 con 0x7f826c0ff990 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.685+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f825c00f5d0 con 0x7f826c0ff990 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f8262ffd700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 0x7f825806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f825c08d450 con 0x7f826c0ff990 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f82712b2700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 0x7f825806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f8262ffd700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 0x7f8258074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8258075490 con 0x7f82580729c0 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f825c08fb20 con 0x7f826c0ff990 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f82712b2700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 0x7f825806f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8268006fd0 tx=0x7f8268008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.686+0000 7f82722b4700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 0x7f8258074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.687+0000 7f82722b4700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 0x7f8258074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.688+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== osd.3 v2:172.21.15.31:6800/2495297239 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8258075490 con 0x7f82580729c0 2024-03-24T10:40:15.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.703+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 --> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f826c1010d0 con 0x7f82580729c0 2024-03-24T10:40:15.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8262ffd700 1 -- 172.21.15.26:0/2910464217 <== osd.3 v2:172.21.15.31:6800/2495297239 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f826c1010d0 con 0x7f82580729c0 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 msgr2=0x7f8258074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 0x7f8258074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 msgr2=0x7f825806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 0x7f825806f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8268006fd0 tx=0x7f8268008040 comp rx=0 tx=0).stop 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 msgr2=0x7f826c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f825c00bd70 tx=0x7f825c00be50 comp rx=0 tx=0).stop 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 shutdown_connections 2024-03-24T10:40:15.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f825806cce0 0x7f825806f1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f826c0ff990 0x7f826c0fbd30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:6800/2495297239,v1:172.21.15.31:6801/2495297239] conn(0x7f82580729c0 0x7f8258074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 --2- 172.21.15.26:0/2910464217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f826c105990 0x7f826c0fc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 >> 172.21.15.26:0/2910464217 conn(0x7f826c0fb4c0 msgr2=0x7f826c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 shutdown_connections 2024-03-24T10:40:15.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.704+0000 7f8273d17700 1 -- 172.21.15.26:0/2910464217 wait complete. 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- 172.21.15.26:0/1321363039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 msgr2=0x7f7d5410ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1321363039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5410ce70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009910 tx=0x7f7d44009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- 172.21.15.26:0/1321363039 shutdown_connections 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1321363039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5410ce70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1321363039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 0x7f7d540ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- 172.21.15.26:0/1321363039 >> 172.21.15.26:0/1321363039 conn(0x7f7d540fb4e0 msgr2=0x7f7d540fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- 172.21.15.26:0/1321363039 shutdown_connections 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- 172.21.15.26:0/1321363039 wait complete. 2024-03-24T10:40:15.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 Processor -- start 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.875+0000 7f7d5a850700 1 -- start start 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 0x7f7d5419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d541989e0 con 0x7f7d541001b0 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54198b50 con 0x7f7d540ff800 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54402/0 (socket says 172.21.15.26:54402) 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 -- 172.21.15.26:0/1965097561 learned_addr learned my addr 172.21.15.26:0/1965097561 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d53fff700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 0x7f7d5419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 -- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 msgr2=0x7f7d5419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 0x7f7d5419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440095f0 con 0x7f7d541001b0 2024-03-24T10:40:15.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d537fe700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7d44006010 tx=0x7f7d440055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4401d070 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d44003d50 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d54198dd0 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4400fe90 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.876+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d54199320 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.877+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d54110180 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f7d4400f6d0 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 0x7f7d3c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d4408d440 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d53fff700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 0x7f7d3c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 0x7f7d3c074c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d3c075320 con 0x7f7d3c072850 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.878+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d4408d8a0 con 0x7f7d541001b0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.879+0000 7f7d58ded700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 0x7f7d3c074c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:15.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.879+0000 7f7d53fff700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 0x7f7d3c06f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005950 tx=0x7f7d4800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.879+0000 7f7d58ded700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 0x7f7d3c074c70 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:15.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.880+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== osd.5 v2:172.21.15.31:6816/2153668083 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d3c075320 con 0x7f7d3c072850 2024-03-24T10:40:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.895+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 --> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7d54199780 con 0x7f7d3c072850 2024-03-24T10:40:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d517fa700 1 -- 172.21.15.26:0/1965097561 <== osd.5 v2:172.21.15.31:6816/2153668083 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7d54199780 con 0x7f7d3c072850 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 msgr2=0x7f7d3c074c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 0x7f7d3c074c70 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 msgr2=0x7f7d3c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 0x7f7d3c06f1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005950 tx=0x7f7d4800b410 comp rx=0 tx=0).stop 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 msgr2=0x7f7d5419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7d44006010 tx=0x7f7d440055f0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 shutdown_connections 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:6816/2153668083,v1:172.21.15.31:6817/2153668083] conn(0x7f7d3c072850 0x7f7d3c074c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f7d3c06cce0 0x7f7d3c06f1a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d540ff800 0x7f7d5419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 --2- 172.21.15.26:0/1965097561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7d541001b0 0x7f7d5419ee70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 >> 172.21.15.26:0/1965097561 conn(0x7f7d540fb4e0 msgr2=0x7f7d540fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 shutdown_connections 2024-03-24T10:40:15.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:15.896+0000 7f7d5a850700 1 -- 172.21.15.26:0/1965097561 wait complete. 2024-03-24T10:40:16.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:16 smithi031 ceph-mon[37345]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:16.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:16 smithi026 ceph-mon[29210]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:115964117012 2024-03-24T10:40:17.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T10:40:18.052 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:17 smithi026 ceph-mon[29210]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:17 smithi031 ceph-mon[37345]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.635+0000 7f922df45700 1 -- 172.21.15.26:0/1573758623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 msgr2=0x7f922810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.635+0000 7f922df45700 1 --2- 172.21.15.26:0/1573758623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922810aa20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f921c009910 tx=0x7f921c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 -- 172.21.15.26:0/1573758623 shutdown_connections 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 --2- 172.21.15.26:0/1573758623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922810aa20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 --2- 172.21.15.26:0/1573758623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 0x7f9228101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 -- 172.21.15.26:0/1573758623 >> 172.21.15.26:0/1573758623 conn(0x7f9228076640 msgr2=0x7f92280fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 -- 172.21.15.26:0/1573758623 shutdown_connections 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 -- 172.21.15.26:0/1573758623 wait complete. 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 Processor -- start 2024-03-24T10:40:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.636+0000 7f922df45700 1 -- start start 2024-03-24T10:40:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 0x7f922819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922819d390 con 0x7f9228102530 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228196970 con 0x7f9228101c10 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54424/0 (socket says 172.21.15.26:54424) 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 -- 172.21.15.26:0/1740862600 learned_addr learned my addr 172.21.15.26:0/1740862600 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f92277fe700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 0x7f922819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 -- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 msgr2=0x7f922819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 0x7f922819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c0095f0 con 0x7f9228102530 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9226ffd700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f921c00bb40 tx=0x7f921c00bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921c01d070 con 0x7f9228102530 2024-03-24T10:40:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9228196bf0 con 0x7f9228102530 2024-03-24T10:40:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f921c003d10 con 0x7f9228102530 2024-03-24T10:40:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921c00fde0 con 0x7f9228102530 2024-03-24T10:40:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.637+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92281970e0 con 0x7f9228102530 2024-03-24T10:40:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.638+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f922810dff0 con 0x7f9228102530 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f921c003850 con 0x7f9228102530 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 0x7f921006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f921c08d2a0 con 0x7f9228102530 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f92277fe700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 0x7f921006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 0x7f9210074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9210075490 con 0x7f92100729c0 2024-03-24T10:40:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.639+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f921c08f8d0 con 0x7f9228102530 2024-03-24T10:40:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.640+0000 7f92277fe700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 0x7f921006f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9218009a10 tx=0x7f9218009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.640+0000 7f9227fff700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 0x7f9210074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.640+0000 7f9227fff700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 0x7f9210074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.641+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== osd.4 v2:172.21.15.31:6808/3345716330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f9210075490 con 0x7f92100729c0 2024-03-24T10:40:18.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.656+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 --> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9228197540 con 0x7f92100729c0 2024-03-24T10:40:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f9224ff9700 1 -- 172.21.15.26:0/1740862600 <== osd.4 v2:172.21.15.31:6808/3345716330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9228197540 con 0x7f92100729c0 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 msgr2=0x7f9210074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 0x7f9210074de0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 msgr2=0x7f921006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 0x7f921006f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9218009a10 tx=0x7f9218009190 comp rx=0 tx=0).stop 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 msgr2=0x7f922819cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f921c00bb40 tx=0x7f921c00bb70 comp rx=0 tx=0).stop 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 shutdown_connections 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:6808/3345716330,v1:172.21.15.31:6809/3345716330] conn(0x7f92100729c0 0x7f9210074de0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f921006cce0 0x7f921006f1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9228101c10 0x7f922819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 --2- 172.21.15.26:0/1740862600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9228102530 0x7f922819cd70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 >> 172.21.15.26:0/1740862600 conn(0x7f9228076640 msgr2=0x7f9228102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 shutdown_connections 2024-03-24T10:40:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:18.657+0000 7f922df45700 1 -- 172.21.15.26:0/1740862600 wait complete. 2024-03-24T10:40:19.258 INFO:teuthology.orchestra.run.smithi026.stdout:171798691849 2024-03-24T10:40:19.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T10:40:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3272213307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc100150 msgr2=0x7f1cdc1005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3272213307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc100150 0x7f1cdc1005d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009910 tx=0x7f1cd0009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3272213307 shutdown_connections 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3272213307 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc100150 0x7f1cdc1005d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3272213307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3272213307 >> 172.21.15.26:0/3272213307 conn(0x7f1cdc075e10 msgr2=0x7f1cdc0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3272213307 shutdown_connections 2024-03-24T10:40:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.739+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3272213307 wait complete. 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 Processor -- start 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 -- start start 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc100150 0x7f1cdc1978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc198400 con 0x7f1cdc1049a0 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1ce1e41700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc198540 con 0x7f1cdc100150 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54446/0 (socket says 172.21.15.26:54446) 2024-03-24T10:40:19.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 -- 172.21.15.26:0/3945728311 learned_addr learned my addr 172.21.15.26:0/3945728311 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdb7fe700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc100150 0x7f1cdc1978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 -- 172.21.15.26:0/3945728311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc100150 msgr2=0x7f1cdc1978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc100150 0x7f1cdc1978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.740+0000 7f1cdaffd700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd00095f0 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1cdaffd700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000bd70 tx=0x7f1cd000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cd001d070 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cd00056e0 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc1a0960 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cd0017400 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.741+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc1a0cc0 con 0x7f1cdc1049a0 2024-03-24T10:40:19.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.742+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cdc10cf10 con 0x7f1cdc1049a0 2024-03-24T10:40:19.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.742+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f1cd0003a60 con 0x7f1cdc1049a0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cd8ff9700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 0x7f1cc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cdb7fe700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 0x7f1cc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1cd008d7c0 con 0x7f1cdc1049a0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cd8ff9700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 0x7f1cc4074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc4075490 con 0x7f1cc40729c0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cd008dc20 con 0x7f1cdc1049a0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cdbfff700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 0x7f1cc4074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cdb7fe700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 0x7f1cc406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005950 tx=0x7f1ccc0058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.743+0000 7f1cdbfff700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 0x7f1cc4074de0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.744+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== osd.1 v2:172.21.15.26:6810/551068105 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1cc4075490 con 0x7f1cc40729c0 2024-03-24T10:40:19.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.759+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 --> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1cdc061430 con 0x7f1cc40729c0 2024-03-24T10:40:19.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.759+0000 7f1cd8ff9700 1 -- 172.21.15.26:0/3945728311 <== osd.1 v2:172.21.15.26:6810/551068105 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1cdc061430 con 0x7f1cc40729c0 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 msgr2=0x7f1cc4074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 0x7f1cc4074de0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 msgr2=0x7f1cc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 0x7f1cc406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005950 tx=0x7f1ccc0058e0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 msgr2=0x7f1cdc197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000bd70 tx=0x7f1cd000be50 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 shutdown_connections 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1cc406cce0 0x7f1cc406f1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:6810/551068105,v1:172.21.15.26:6811/551068105] conn(0x7f1cc40729c0 0x7f1cc4074de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cdc100150 0x7f1cdc1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 --2- 172.21.15.26:0/3945728311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cdc1049a0 0x7f1cdc197e30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 >> 172.21.15.26:0/3945728311 conn(0x7f1cdc075e10 msgr2=0x7f1cdc0fb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 shutdown_connections 2024-03-24T10:40:19.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:19.760+0000 7f1ce1e41700 1 -- 172.21.15.26:0/3945728311 wait complete. 2024-03-24T10:40:19.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:19 smithi026 ceph-mon[29210]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:20.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:19 smithi031 ceph-mon[37345]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:20.473 INFO:teuthology.orchestra.run.smithi026.stdout:60129542175 2024-03-24T10:40:20.473 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T10:40:20.923 INFO:teuthology.orchestra.run.smithi026.stdout:38654705699 2024-03-24T10:40:20.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T10:40:21.696 INFO:teuthology.orchestra.run.smithi026.stdout:150323855374 2024-03-24T10:40:21.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T10:40:21.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:21 smithi026 ceph-mon[29210]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:22.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:21 smithi031 ceph-mon[37345]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/1010844431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 msgr2=0x7fe2dc105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/1010844431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc105d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00b0a0 tx=0x7fe2cc00b3b0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/1010844431 shutdown_connections 2024-03-24T10:40:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/1010844431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 0x7fe2dc0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/1010844431 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc105d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/1010844431 >> 172.21.15.26:0/1010844431 conn(0x7fe2dc0fb480 msgr2=0x7fe2dc0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/1010844431 shutdown_connections 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.666+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/1010844431 wait complete. 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 Processor -- start 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 -- start start 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 0x7fe2dc197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc198360 con 0x7fe2dc105970 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e3ee0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1984a0 con 0x7fe2dc0ff970 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55402/0 (socket says 172.21.15.26:55402) 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 -- 172.21.15.26:0/374439845 learned_addr learned my addr 172.21.15.26:0/374439845 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e1c7c700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 0x7fe2dc197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 -- 172.21.15.26:0/374439845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 msgr2=0x7fe2dc197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 0x7fe2dc197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.667+0000 7fe2e147b700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc009d00 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2e147b700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000d8d0 tx=0x7fe2d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2d00098a0 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2d0010460 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc1a08c0 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2d000f5f0 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.668+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc1a0cd0 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.669+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2dc10ce70 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.669+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fe2d0009a00 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.669+0000 7fe2daffd700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 0x7fe2c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe2d008bd20 con 0x7fe2dc105970 2024-03-24T10:40:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2e1c7c700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 0x7fe2c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2daffd700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 0x7fe2c4074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2c4075490 con 0x7fe2c40729c0 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2d0091400 con 0x7fe2dc105970 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2e247d700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 0x7fe2c4074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2e1c7c700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 0x7fe2c406f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000f80 tx=0x7fe2cc0060c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.670+0000 7fe2e247d700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 0x7fe2c4074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.671+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== osd.2 v2:172.21.15.26:6818/291489572 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2c4075490 con 0x7fe2c40729c0 2024-03-24T10:40:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 --> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2dc0614f0 con 0x7fe2c40729c0 2024-03-24T10:40:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2daffd700 1 -- 172.21.15.26:0/374439845 <== osd.2 v2:172.21.15.26:6818/291489572 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2dc0614f0 con 0x7fe2c40729c0 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 msgr2=0x7fe2c4074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 0x7fe2c4074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 msgr2=0x7fe2c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 0x7fe2c406f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000f80 tx=0x7fe2cc0060c0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 msgr2=0x7fe2dc197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000d8d0 tx=0x7fe2d000dbe0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 shutdown_connections 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6818/291489572,v1:172.21.15.26:6819/291489572] conn(0x7fe2c40729c0 0x7fe2c4074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe2c406cce0 0x7fe2c406f1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0ff970 0x7fe2dc197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 --2- 172.21.15.26:0/374439845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2dc105970 0x7fe2dc197d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 >> 172.21.15.26:0/374439845 conn(0x7fe2dc0fb480 msgr2=0x7fe2dc0fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 shutdown_connections 2024-03-24T10:40:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:22.687+0000 7fe2e3ee0700 1 -- 172.21.15.26:0/374439845 wait complete. 2024-03-24T10:40:23.488 INFO:teuthology.orchestra.run.smithi026.stdout:81604378650 2024-03-24T10:40:23.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T10:40:23.510 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:23.510 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:23.568 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:23.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:23 smithi026 ceph-mon[29210]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:24.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:23 smithi031 ceph-mon[37345]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:26.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:26 smithi031 ceph-mon[37345]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:26 smithi026 ceph-mon[29210]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:27.348 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:28.026 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:27 smithi031 ceph-mon[37345]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:28.075 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:27 smithi026 ceph-mon[29210]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:29.043 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:28 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:29.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:28 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:30.074 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:29 smithi026 ceph-mon[29210]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 -- 172.21.15.26:0/3247928554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 msgr2=0x7effac0ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 --2- 172.21.15.26:0/3247928554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac0ffdd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7effa0009910 tx=0x7effa0009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 -- 172.21.15.26:0/3247928554 shutdown_connections 2024-03-24T10:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 --2- 172.21.15.26:0/3247928554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac0ffdd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 --2- 172.21.15.26:0/3247928554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 0x7effac105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 -- 172.21.15.26:0/3247928554 >> 172.21.15.26:0/3247928554 conn(0x7effac076640 msgr2=0x7effac0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 -- 172.21.15.26:0/3247928554 shutdown_connections 2024-03-24T10:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.103+0000 7effb0bf3700 1 -- 172.21.15.26:0/3247928554 wait complete. 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 Processor -- start 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 -- start start 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 0x7effac19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac19d280 con 0x7effac0ff950 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.104+0000 7effb0bf3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac1967f0 con 0x7effac105950 2024-03-24T10:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55416/0 (socket says 172.21.15.26:55416) 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 -- 172.21.15.26:0/638470132 learned_addr learned my addr 172.21.15.26:0/638470132 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effa9d9b700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 0x7effac19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 -- 172.21.15.26:0/638470132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 msgr2=0x7effac19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 0x7effac19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 -- 172.21.15.26:0/638470132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa00095f0 con 0x7effac0ff950 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effaa59c700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e920 tx=0x7eff9c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c00c9d0 con 0x7effac0ff950 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff9c010430 con 0x7effac0ff950 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac196ad0 con 0x7effac0ff950 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c018630 con 0x7effac0ff950 2024-03-24T10:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.105+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac197020 con 0x7effac0ff950 2024-03-24T10:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.106+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac101090 con 0x7effac0ff950 2024-03-24T10:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.107+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7eff9c0105a0 con 0x7effac0ff950 2024-03-24T10:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.107+0000 7eff9b7fe700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 0x7eff9406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.107+0000 7effa9d9b700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 0x7eff9406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.107+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7eff9c014070 con 0x7effac0ff950 2024-03-24T10:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.108+0000 7effa9d9b700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 0x7eff9406f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7effac197c10 tx=0x7effa0005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.112+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff9c0529f0 con 0x7effac0ff950 2024-03-24T10:40:30.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:29 smithi031 ceph-mon[37345]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:30.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.318+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7effac04ea80 con 0x7effac0ff950 2024-03-24T10:40:30.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.318+0000 7eff9b7fe700 1 -- 172.21.15.26:0/638470132 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff9c05a350 con 0x7effac0ff950 2024-03-24T10:40:30.319 INFO:teuthology.orchestra.run.smithi026.stdout:38654705701 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 msgr2=0x7eff9406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 0x7eff9406f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7effac197c10 tx=0x7effa0005940 comp rx=0 tx=0).stop 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 msgr2=0x7effac19c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e920 tx=0x7eff9c00ec30 comp rx=0 tx=0).stop 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 shutdown_connections 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7eff9406cce0 0x7eff9406f1a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effac0ff950 0x7effac19c6b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 --2- 172.21.15.26:0/638470132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effac105950 0x7effac19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 >> 172.21.15.26:0/638470132 conn(0x7effac076640 msgr2=0x7effac104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 shutdown_connections 2024-03-24T10:40:30.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:30.320+0000 7effb0bf3700 1 -- 172.21.15.26:0/638470132 wait complete. 2024-03-24T10:40:31.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-03-24T10:40:31.014 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:31.246 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:31 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/638470132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T10:40:31.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:31 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/638470132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T10:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 -- 172.21.15.26:0/4198824428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80730d0 msgr2=0x7f61d80734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 --2- 172.21.15.26:0/4198824428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80730d0 0x7f61d80734b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009910 tx=0x7f61c8009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 -- 172.21.15.26:0/4198824428 shutdown_connections 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 --2- 172.21.15.26:0/4198824428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80739f0 0x7f61d810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 --2- 172.21.15.26:0/4198824428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80730d0 0x7f61d80734b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 -- 172.21.15.26:0/4198824428 >> 172.21.15.26:0/4198824428 conn(0x7f61d80fc5a0 msgr2=0x7f61d80fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 -- 172.21.15.26:0/4198824428 shutdown_connections 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.974+0000 7f61dd746700 1 -- 172.21.15.26:0/4198824428 wait complete. 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 Processor -- start 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 -- start start 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80730d0 0x7f61d819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d81989f0 con 0x7f61d80739f0 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61dd746700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d8198b60 con 0x7f61d80730d0 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61d67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61d67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55442/0 (socket says 172.21.15.26:55442) 2024-03-24T10:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61d67fc700 1 -- 172.21.15.26:0/1486170848 learned_addr learned my addr 172.21.15.26:0/1486170848 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.975+0000 7f61d6ffd700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80730d0 0x7f61d819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61d67fc700 1 -- 172.21.15.26:0/1486170848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80730d0 msgr2=0x7f61d819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61d67fc700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80730d0 0x7f61d819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61d67fc700 1 -- 172.21.15.26:0/1486170848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80095f0 con 0x7f61d80739f0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61d67fc700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00d8d0 tx=0x7f61cc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61cc0098a0 con 0x7f61d80739f0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61cc010460 con 0x7f61d80739f0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d8198e40 con 0x7f61d80739f0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61cc00f5f0 con 0x7f61d80739f0 2024-03-24T10:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.976+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d8199390 con 0x7f61d80739f0 2024-03-24T10:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.977+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d8074810 con 0x7f61d80739f0 2024-03-24T10:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.978+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f61cc009a00 con 0x7f61d80739f0 2024-03-24T10:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.978+0000 7f61bffff700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 0x7f61c006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.979+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f61cc08bd20 con 0x7f61d80739f0 2024-03-24T10:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.979+0000 7f61d6ffd700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 0x7f61c006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.979+0000 7f61d6ffd700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 0x7f61c006f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f61c8006010 tx=0x7f61c801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:31.983+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61cc0564b0 con 0x7f61d80739f0 2024-03-24T10:40:32.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.193+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f61d8107600 con 0x7f61d80739f0 2024-03-24T10:40:32.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.194+0000 7f61bffff700 1 -- 172.21.15.26:0/1486170848 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61cc059ad0 con 0x7f61d80739f0 2024-03-24T10:40:32.194 INFO:teuthology.orchestra.run.smithi026.stdout:60129542177 2024-03-24T10:40:32.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 msgr2=0x7f61c006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 0x7f61c006f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f61c8006010 tx=0x7f61c801a040 comp rx=0 tx=0).stop 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 msgr2=0x7f61d819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00d8d0 tx=0x7f61cc00dbe0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 shutdown_connections 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f61c006cce0 0x7f61c006f1a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d80730d0 0x7f61d819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 --2- 172.21.15.26:0/1486170848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61d80739f0 0x7f61d819ee80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 >> 172.21.15.26:0/1486170848 conn(0x7f61d80fc5a0 msgr2=0x7f61d81052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 shutdown_connections 2024-03-24T10:40:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.196+0000 7f61dd746700 1 -- 172.21.15.26:0/1486170848 wait complete. 2024-03-24T10:40:32.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:32 smithi026 ceph-mon[29210]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2622977237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b680ff7e0 msgr2=0x7f8b680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2622977237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b680ff7e0 0x7f8b680ffbc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009910 tx=0x7f8b58009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2622977237 shutdown_connections 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2622977237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b68100190 0x7f8b6810ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2622977237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b680ff7e0 0x7f8b680ffbc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2622977237 >> 172.21.15.26:0/2622977237 conn(0x7f8b680fb4c0 msgr2=0x7f8b680fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2622977237 shutdown_connections 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2622977237 wait complete. 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 Processor -- start 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.279+0000 7f8b6e8c5700 1 -- start start 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b680ff7e0 0x7f8b6819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68198a70 con 0x7f8b68100190 2024-03-24T10:40:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68198be0 con 0x7f8b680ff7e0 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55460/0 (socket says 172.21.15.26:55460) 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 -- 172.21.15.26:0/2480110033 learned_addr learned my addr 172.21.15.26:0/2480110033 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b67fff700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b680ff7e0 0x7f8b6819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 -- 172.21.15.26:0/2480110033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b680ff7e0 msgr2=0x7f8b6819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b680ff7e0 0x7f8b6819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 -- 172.21.15.26:0/2480110033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580095f0 con 0x7f8b68100190 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b677fe700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c8a0 tx=0x7f8b5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5c00cea0 con 0x7f8b68100190 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b68198ec0 con 0x7f8b68100190 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b5c004830 con 0x7f8b68100190 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5c0055b0 con 0x7f8b68100190 2024-03-24T10:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.280+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b68199410 con 0x7f8b68100190 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.281+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6810acd0 con 0x7f8b68100190 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.282+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f8b5c0049a0 con 0x7f8b68100190 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.282+0000 7f8b657fa700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 0x7f8b5006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.283+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8b5c08bbb0 con 0x7f8b68100190 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.283+0000 7f8b67fff700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 0x7f8b5006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.283+0000 7f8b67fff700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 0x7f8b5006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8b58000c00 tx=0x7f8b5800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.288+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b5c092080 con 0x7f8b68100190 2024-03-24T10:40:32.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:32 smithi031 ceph-mon[37345]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:32.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.492+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8b68199f50 con 0x7f8b68100190 2024-03-24T10:40:32.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.492+0000 7f8b657fa700 1 -- 172.21.15.26:0/2480110033 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8b5c0562c0 con 0x7f8b68100190 2024-03-24T10:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout:81604378652 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 msgr2=0x7f8b5006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 0x7f8b5006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8b58000c00 tx=0x7f8b5800b540 comp rx=0 tx=0).stop 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 msgr2=0x7f8b6819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c8a0 tx=0x7f8b5c00cc60 comp rx=0 tx=0).stop 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 shutdown_connections 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8b5006cce0 0x7f8b5006f1a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b680ff7e0 0x7f8b6819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.494+0000 7f8b6e8c5700 1 --2- 172.21.15.26:0/2480110033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b68100190 0x7f8b6819ef00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.495+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 >> 172.21.15.26:0/2480110033 conn(0x7f8b680fb4c0 msgr2=0x7f8b68107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:32.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.495+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 shutdown_connections 2024-03-24T10:40:32.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.495+0000 7f8b6e8c5700 1 -- 172.21.15.26:0/2480110033 wait complete. 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 -- 172.21.15.26:0/2507676370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 msgr2=0x7fe3f0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 --2- 172.21.15.26:0/2507676370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f0106e90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009910 tx=0x7fe3ec009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 -- 172.21.15.26:0/2507676370 shutdown_connections 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 --2- 172.21.15.26:0/2507676370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 0x7fe3f0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 --2- 172.21.15.26:0/2507676370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f0106e90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 -- 172.21.15.26:0/2507676370 >> 172.21.15.26:0/2507676370 conn(0x7fe3f00fc5e0 msgr2=0x7fe3f00fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 -- 172.21.15.26:0/2507676370 shutdown_connections 2024-03-24T10:40:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.703+0000 7fe3f77de700 1 -- 172.21.15.26:0/2507676370 wait complete. 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 Processor -- start 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 -- start start 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 0x7fe3f019c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f019d220 con 0x7fe3f0106ab0 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.704+0000 7fe3f77de700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01967d0 con 0x7fe3f0100ab0 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36918/0 (socket says 172.21.15.26:36918) 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 -- 172.21.15.26:0/387924728 learned_addr learned my addr 172.21.15.26:0/387924728 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f557a700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 0x7fe3f019c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 -- 172.21.15.26:0/387924728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 msgr2=0x7fe3f019c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 0x7fe3f019c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 -- 172.21.15.26:0/387924728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec0095f0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f4d79700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000d8d0 tx=0x7fe3e000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e00098a0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3e0010460 con 0x7fe3f0106ab0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f0196ab0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e000f5f0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.705+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f0197000 con 0x7fe3f0106ab0 2024-03-24T10:40:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.706+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f01090b0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.707+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fe3e0009a00 con 0x7fe3f0106ab0 2024-03-24T10:40:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.707+0000 7fe3e67fc700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 0x7fe3dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.707+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe3e008bdc0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.707+0000 7fe3f557a700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 0x7fe3dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:32.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.708+0000 7fe3f557a700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 0x7fe3dc06f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec000c00 tx=0x7fe3ec01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:32.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.712+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3e0059f60 con 0x7fe3f0106ab0 2024-03-24T10:40:32.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.918+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe3f0108a90 con 0x7fe3f0106ab0 2024-03-24T10:40:32.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.918+0000 7fe3e67fc700 1 -- 172.21.15.26:0/387924728 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe3e0059af0 con 0x7fe3f0106ab0 2024-03-24T10:40:32.919 INFO:teuthology.orchestra.run.smithi026.stdout:115964117015 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.920+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 msgr2=0x7fe3dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.920+0000 7fe3f77de700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 0x7fe3dc06f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec000c00 tx=0x7fe3ec01a040 comp rx=0 tx=0).stop 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.920+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 msgr2=0x7fe3f019cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.920+0000 7fe3f77de700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000d8d0 tx=0x7fe3e000dbe0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 shutdown_connections 2024-03-24T10:40:32.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fe3dc06cce0 0x7fe3dc06f1a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3f0100ab0 0x7fe3f019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 --2- 172.21.15.26:0/387924728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f0106ab0 0x7fe3f019cb40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:32.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 >> 172.21.15.26:0/387924728 conn(0x7fe3f00fc5e0 msgr2=0x7fe3f00fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:32.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 shutdown_connections 2024-03-24T10:40:32.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.921+0000 7fe3f77de700 1 -- 172.21.15.26:0/387924728 wait complete. 2024-03-24T10:40:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 -- 172.21.15.26:0/471790098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 msgr2=0x7fde98105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 --2- 172.21.15.26:0/471790098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde98105bc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009910 tx=0x7fde8c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 -- 172.21.15.26:0/471790098 shutdown_connections 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 --2- 172.21.15.26:0/471790098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 0x7fde980ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 --2- 172.21.15.26:0/471790098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde98105bc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 -- 172.21.15.26:0/471790098 >> 172.21.15.26:0/471790098 conn(0x7fde98075370 msgr2=0x7fde98075780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 -- 172.21.15.26:0/471790098 shutdown_connections 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.915+0000 7fde9db9d700 1 -- 172.21.15.26:0/471790098 wait complete. 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 Processor -- start 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 -- start start 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 0x7fde9819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98196650 con 0x7fde981057e0 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9db9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981967c0 con 0x7fde980ff8f0 2024-03-24T10:40:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36930/0 (socket says 172.21.15.26:36930) 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde97fff700 1 -- 172.21.15.26:0/3165572082 learned_addr learned my addr 172.21.15.26:0/3165572082 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde9cb9b700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 0x7fde9819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde97fff700 1 -- 172.21.15.26:0/3165572082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 msgr2=0x7fde9819c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stdout:171798691853 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.916+0000 7fde97fff700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 0x7fde9819c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde97fff700 1 -- 172.21.15.26:0/3165572082 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0095f0 con 0x7fde981057e0 2024-03-24T10:40:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde97fff700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fde8400bf20 tx=0x7fde8400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde84012c60 con 0x7fde981057e0 2024-03-24T10:40:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde84016070 con 0x7fde981057e0 2024-03-24T10:40:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde98196a40 con 0x7fde981057e0 2024-03-24T10:40:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde84009790 con 0x7fde981057e0 2024-03-24T10:40:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.917+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde98196f90 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.918+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde98100f20 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.919+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fde840098f0 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.919+0000 7fde95ffb700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 0x7fde8806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.919+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fde84018070 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.919+0000 7fde9cb9b700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 0x7fde8806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.920+0000 7fde9cb9b700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 0x7fde8806f1b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde8c006010 tx=0x7fde8c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:32.924+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde8405a630 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.140+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fde98197bd0 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.140+0000 7fde95ffb700 1 -- 172.21.15.26:0/3165572082 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fde8405dc50 con 0x7fde981057e0 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.142+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 msgr2=0x7fde8806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.142+0000 7fde9db9d700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 0x7fde8806f1b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde8c006010 tx=0x7fde8c01a040 comp rx=0 tx=0).stop 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 msgr2=0x7fde9819cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fde8400bf20 tx=0x7fde8400dbb0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 shutdown_connections 2024-03-24T10:40:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fde8806ccf0 0x7fde8806f1b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde980ff8f0 0x7fde9819c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 --2- 172.21.15.26:0/3165572082 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde981057e0 0x7fde9819cae0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 >> 172.21.15.26:0/3165572082 conn(0x7fde98075370 msgr2=0x7fde980687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 shutdown_connections 2024-03-24T10:40:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.143+0000 7fde9db9d700 1 -- 172.21.15.26:0/3165572082 wait complete. 2024-03-24T10:40:33.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:33 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1486170848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T10:40:33.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:33 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2480110033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T10:40:33.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:33 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/387924728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T10:40:33.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:33 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3165572082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T10:40:33.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1486170848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T10:40:33.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2480110033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T10:40:33.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/387924728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T10:40:33.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3165572082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T10:40:33.934 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117015 for osd.3 2024-03-24T10:40:33.934 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:34.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:34 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3733413042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T10:40:34.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:34 smithi031 ceph-mon[37345]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:34.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:34 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3733413042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T10:40:34.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:34 smithi026 ceph-mon[29210]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:34.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378650 got 81604378652 for osd.2 2024-03-24T10:40:34.616 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:35.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691853 for osd.5 2024-03-24T10:40:35.000 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:35.424 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-03-24T10:40:35.424 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3921545462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4105970 msgr2=0x7ff8b4105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3921545462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4105970 0x7ff8b4105d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009910 tx=0x7ff8b0009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3921545462 shutdown_connections 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3921545462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b40ff970 0x7ff8b40ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3921545462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4105970 0x7ff8b4105d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3921545462 >> 172.21.15.26:0/3921545462 conn(0x7ff8b40fb480 msgr2=0x7ff8b40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3921545462 shutdown_connections 2024-03-24T10:40:35.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.041+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3921545462 wait complete. 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 Processor -- start 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 -- start start 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b4105970 0x7ff8b419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b419c150 con 0x7ff8b40ff970 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8bb6a3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4195730 con 0x7ff8b4105970 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36940/0 (socket says 172.21.15.26:36940) 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 -- 172.21.15.26:0/3733413042 learned_addr learned my addr 172.21.15.26:0/3733413042 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b8c3e700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b4105970 0x7ff8b419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 -- 172.21.15.26:0/3733413042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b4105970 msgr2=0x7ff8b419bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b4105970 0x7ff8b419bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.042+0000 7ff8b943f700 1 -- 172.21.15.26:0/3733413042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b00095f0 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8b943f700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b001d070 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8b00056e0 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b41959b0 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b000fe80 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.043+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b4195ea0 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.044+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b4107f70 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.044+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ff8b0003a60 con 0x7ff8b40ff970 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.044+0000 7ff8aa7fc700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 0x7ff8a006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:35.578 INFO:teuthology.orchestra.run.smithi026.stdout:150323855377 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.045+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff8b008d290 con 0x7ff8b40ff970 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.045+0000 7ff8b8c3e700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 0x7ff8a006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.045+0000 7ff8b8c3e700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 0x7ff8a006f1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4005950 tx=0x7ff8a40058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.049+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff8b005b4b0 con 0x7ff8b40ff970 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.256+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff8b4196bf0 con 0x7ff8b40ff970 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.256+0000 7ff8aa7fc700 1 -- 172.21.15.26:0/3733413042 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff8b005b040 con 0x7ff8b40ff970 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 msgr2=0x7ff8a006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 0x7ff8a006f1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4005950 tx=0x7ff8a40058e0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 msgr2=0x7ff8b419b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b000be20 comp rx=0 tx=0).stop 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 shutdown_connections 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff8a006ccf0 0x7ff8a006f1b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b40ff970 0x7ff8b419b5f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 --2- 172.21.15.26:0/3733413042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8b4105970 0x7ff8b419bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 >> 172.21.15.26:0/3733413042 conn(0x7ff8b40fb480 msgr2=0x7ff8b40fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 shutdown_connections 2024-03-24T10:40:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:33.258+0000 7ff8bb6a3700 1 -- 172.21.15.26:0/3733413042 wait complete. 2024-03-24T10:40:36.131 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855374 got 150323855377 for osd.4 2024-03-24T10:40:36.131 DEBUG:teuthology.parallel:result is None 2024-03-24T10:40:36.131 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T10:40:36.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T10:40:36.354 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:36.396 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:36 smithi026 ceph-mon[29210]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:36.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:36 smithi031 ceph-mon[37345]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:38.081 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:37 smithi026 ceph-mon[29210]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:38.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- 172.21.15.26:0/1646254037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684106a70 msgr2=0x7f5684106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:38.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 --2- 172.21.15.26:0/1646254037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684106a70 0x7f5684106e50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5680009910 tx=0x7f5680009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:38.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- 172.21.15.26:0/1646254037 shutdown_connections 2024-03-24T10:40:38.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 --2- 172.21.15.26:0/1646254037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684100a70 0x7f5684100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 --2- 172.21.15.26:0/1646254037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684106a70 0x7f5684106e50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- 172.21.15.26:0/1646254037 >> 172.21.15.26:0/1646254037 conn(0x7f56840fc5a0 msgr2=0x7f56840fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:38.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- 172.21.15.26:0/1646254037 shutdown_connections 2024-03-24T10:40:38.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- 172.21.15.26:0/1646254037 wait complete. 2024-03-24T10:40:38.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 Processor -- start 2024-03-24T10:40:38.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.184+0000 7f568be8d700 1 -- start start 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684106a70 0x7f568419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568419f4e0 con 0x7f5684100a70 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5684198ac0 con 0x7f5684106a70 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36952/0 (socket says 172.21.15.26:36952) 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 -- 172.21.15.26:0/1617102221 learned_addr learned my addr 172.21.15.26:0/1617102221 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689428700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684106a70 0x7f568419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 -- 172.21.15.26:0/1617102221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684106a70 msgr2=0x7f568419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684106a70 0x7f568419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 -- 172.21.15.26:0/1617102221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800095f0 con 0x7f5684100a70 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f5689c29700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5680006010 tx=0x7f568000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568001d070 con 0x7f5684100a70 2024-03-24T10:40:38.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f568000f460 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5684198d40 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5680017630 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.185+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5684199230 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.186+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5684104d20 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.187+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f568000f5d0 con 0x7f5684100a70 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.187+0000 7f567affd700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 0x7f567006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.187+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f568008d2f0 con 0x7f5684100a70 2024-03-24T10:40:38.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.187+0000 7f5689428700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 0x7f567006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:38.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.188+0000 7f5689428700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 0x7f567006f1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5684199c80 tx=0x7f5674009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.192+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f568005b5d0 con 0x7f5684100a70 2024-03-24T10:40:38.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:37 smithi031 ceph-mon[37345]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.388+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5684199f80 con 0x7f567006ccf0 2024-03-24T10:40:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.391+0000 7f567affd700 1 -- 172.21.15.26:0/1617102221 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27755 (secure 0 0 0) 0x7f5684199f80 con 0x7f567006ccf0 2024-03-24T10:40:38.392 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.393+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 msgr2=0x7f567006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.393+0000 7f568be8d700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 0x7f567006f1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5684199c80 tx=0x7f5674009190 comp rx=0 tx=0).stop 2024-03-24T10:40:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.393+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 msgr2=0x7f568419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.393+0000 7f568be8d700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5680006010 tx=0x7f568000bd70 comp rx=0 tx=0).stop 2024-03-24T10:40:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 shutdown_connections 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f567006ccf0 0x7f567006f1b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5684100a70 0x7f568419e980 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 --2- 172.21.15.26:0/1617102221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684106a70 0x7f568419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 >> 172.21.15.26:0/1617102221 conn(0x7f56840fc5a0 msgr2=0x7f5684073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 shutdown_connections 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:38.394+0000 7f568be8d700 1 -- 172.21.15.26:0/1617102221 wait complete. 2024-03-24T10:40:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-24T10:40:38.974 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-03-24T10:40:37.686225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166056,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":562304856,"statfs":{"total":575970213888,"available":575800172544,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-24T10:39:45.365377+0000","last_change":"2024-03-24T10:39:19.769630+0000","last_active":"2024-03-24T10:39:45.365377+0000","last_peered":"2024-03-24T10:39:45.365377+0000","last_clean":"2024-03-24T10:39:45.365377+0000","last_became_active":"2024-03-24T10:39:19.769265+0000","last_became_peered":"2024-03-24T10:39:19.769265+0000","last_unstale":"2024-03-24T10:39:45.365377+0000","last_undegraded":"2024-03-24T10:39:45.365377+0000","last_fullsized":"2024-03-24T10:39:45.365377+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:32.681061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:32.681061+0000","last_clean_scrub_stamp":"2024-03-24T10:38:32.681061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:57:06.950805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.64300000000000002}]},{"osd":2,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.435},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.45100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.59199999999999997}]},{"osd":1,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.71199999999999997}]},{"osd":2,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.66300000000000003}]},{"osd":4,"last update":"Sun Mar 24 10:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.215,"5min":0.19,"15min":0.19},"max":{"1min":0.44700000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.46800000000000003,"5min":0.495,"15min":0.495},"last":0.33600000000000002}]},{"osd":1,"last update":"Sun Mar 24 10:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.437,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.36599999999999999,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49099999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.35699999999999998}]},{"osd":3,"last update":"Sun Mar 24 10:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.623}]},{"osd":4,"last update":"Sun Mar 24 10:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 10:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.31},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.32600000000000001},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Mar 24 10:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Sun Mar 24 10:39:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.496},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51600000000000001}]},{"osd":4,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33400000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.33600000000000002,"15min":0.33100000000000002},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"last":0.23599999999999999}]},{"osd":2,"last update":"Sun Mar 24 10:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.30599999999999999}]},{"osd":3,"last update":"Sun Mar 24 10:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.502},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.44400000000000001}]},{"osd":4,"last update":"Sun Mar 24 10:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.496},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:40:38.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T10:40:39.183 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:39.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:38 smithi031 ceph-mon[37345]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:39.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:38 smithi026 ceph-mon[29210]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:39.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:39 smithi026 ceph-mon[29210]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:40.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:39 smithi031 ceph-mon[37345]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:40.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 -- 172.21.15.26:0/4252969164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 msgr2=0x7fdbf0105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/4252969164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0105d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009910 tx=0x7fdbe0009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 -- 172.21.15.26:0/4252969164 shutdown_connections 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/4252969164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 0x7fdbf00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/4252969164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0105d80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 -- 172.21.15.26:0/4252969164 >> 172.21.15.26:0/4252969164 conn(0x7fdbf00fb4b0 msgr2=0x7fdbf00fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 -- 172.21.15.26:0/4252969164 shutdown_connections 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.762+0000 7fdbf63b2700 1 -- 172.21.15.26:0/4252969164 wait complete. 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 Processor -- start 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 -- start start 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 0x7fdbf0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0198450 con 0x7fdbf01059a0 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.763+0000 7fdbf63b2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0198590 con 0x7fdbf00ff9a0 2024-03-24T10:40:40.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36978/0 (socket says 172.21.15.26:36978) 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 -- 172.21.15.26:0/344722365 learned_addr learned my addr 172.21.15.26:0/344722365 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbeffff700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 0x7fdbf0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 -- 172.21.15.26:0/344722365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 msgr2=0x7fdbf0197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 0x7fdbf0197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 -- 172.21.15.26:0/344722365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00095f0 con 0x7fdbf01059a0 2024-03-24T10:40:40.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbef7fe700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b4b0 tx=0x7fdbd800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbd8011800 con 0x7fdbf01059a0 2024-03-24T10:40:40.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf01a09b0 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbd8011e20 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbd800f2e0 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.764+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf01a0dc0 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.765+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0107fa0 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.766+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fdbd8010380 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.766+0000 7fdbed7fa700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 0x7fdbdc06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.766+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdbd808ba60 con 0x7fdbf01059a0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.766+0000 7fdbeffff700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 0x7fdbdc06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.767+0000 7fdbeffff700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 0x7fdbdc06f1b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0006010 tx=0x7fdbe001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.771+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdbd8056170 con 0x7fdbf01059a0 2024-03-24T10:40:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.967+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbf0069600 con 0x7fdbdc06ccf0 2024-03-24T10:40:40.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.968+0000 7fdbed7fa700 1 -- 172.21.15.26:0/344722365 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27811 (secure 0 0 0) 0x7fdbf0069600 con 0x7fdbdc06ccf0 2024-03-24T10:40:40.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:40.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 msgr2=0x7fdbdc06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 0x7fdbdc06f1b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0006010 tx=0x7fdbe001a040 comp rx=0 tx=0).stop 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 msgr2=0x7fdbf0197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b4b0 tx=0x7fdbd800b7c0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 shutdown_connections 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdbdc06ccf0 0x7fdbdc06f1b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdbf00ff9a0 0x7fdbf0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 --2- 172.21.15.26:0/344722365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf01059a0 0x7fdbf0197e80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 >> 172.21.15.26:0/344722365 conn(0x7fdbf00fb4b0 msgr2=0x7fdbf00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 shutdown_connections 2024-03-24T10:40:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:40.971+0000 7fdbf63b2700 1 -- 172.21.15.26:0/344722365 wait complete. 2024-03-24T10:40:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-24T10:40:41.546 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-03-24T10:40:39.686572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166056,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":562304856,"statfs":{"total":575970213888,"available":575800172544,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002444"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-24T10:39:45.365377+0000","last_change":"2024-03-24T10:39:19.769630+0000","last_active":"2024-03-24T10:39:45.365377+0000","last_peered":"2024-03-24T10:39:45.365377+0000","last_clean":"2024-03-24T10:39:45.365377+0000","last_became_active":"2024-03-24T10:39:19.769265+0000","last_became_peered":"2024-03-24T10:39:19.769265+0000","last_unstale":"2024-03-24T10:39:45.365377+0000","last_undegraded":"2024-03-24T10:39:45.365377+0000","last_fullsized":"2024-03-24T10:39:45.365377+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:32.681061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:32.681061+0000","last_clean_scrub_stamp":"2024-03-24T10:38:32.681061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:57:06.950805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.64300000000000002}]},{"osd":2,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.435},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.45100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.59199999999999997}]},{"osd":1,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.71199999999999997}]},{"osd":2,"last update":"Sun Mar 24 10:39:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.66300000000000003}]},{"osd":4,"last update":"Sun Mar 24 10:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.215,"5min":0.19,"15min":0.19},"max":{"1min":0.44700000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.46800000000000003,"5min":0.495,"15min":0.495},"last":0.33600000000000002}]},{"osd":1,"last update":"Sun Mar 24 10:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.437,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.36599999999999999,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49099999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.35699999999999998}]},{"osd":3,"last update":"Sun Mar 24 10:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.623}]},{"osd":4,"last update":"Sun Mar 24 10:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 10:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.31},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.32600000000000001},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Mar 24 10:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Sun Mar 24 10:39:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.496},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51600000000000001}]},{"osd":4,"last update":"Sun Mar 24 10:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 10:40:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33400000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.33600000000000002,"15min":0.33100000000000002},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"last":0.36899999999999999}]},{"osd":2,"last update":"Sun Mar 24 10:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.35599999999999998,"15min":0.35799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.372,"15min":0.376},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.47299999999999998,"5min":0.496,"15min":0.496},"last":0.27200000000000002}]},{"osd":3,"last update":"Sun Mar 24 10:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64100000000000001}]},{"osd":4,"last update":"Sun Mar 24 10:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:40:41.547 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T10:40:41.547 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T10:40:41.547 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T10:40:41.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T10:40:41.754 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:41.876 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:41 smithi026 ceph-mon[29210]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:41.876 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:41 smithi026 ceph-mon[29210]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:42.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:41 smithi031 ceph-mon[37345]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:42.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:41 smithi031 ceph-mon[37345]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 -- 172.21.15.26:0/451372944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 msgr2=0x7f70800ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 --2- 172.21.15.26:0/451372944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70800ffdf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7074009910 tx=0x7f7074009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 -- 172.21.15.26:0/451372944 shutdown_connections 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 --2- 172.21.15.26:0/451372944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70800ffdf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 --2- 172.21.15.26:0/451372944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 -- 172.21.15.26:0/451372944 >> 172.21.15.26:0/451372944 conn(0x7f70800fb480 msgr2=0x7f70800fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 -- 172.21.15.26:0/451372944 shutdown_connections 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.334+0000 7f7087d27700 1 -- 172.21.15.26:0/451372944 wait complete. 2024-03-24T10:40:43.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 Processor -- start 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 -- start start 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080198300 con 0x7f70800ff970 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7087d27700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080198440 con 0x7f7080105970 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59414/0 (socket says 172.21.15.26:59414) 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 -- 172.21.15.26:0/3252202344 learned_addr learned my addr 172.21.15.26:0/3252202344 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f70852c2700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 -- 172.21.15.26:0/3252202344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 msgr2=0x7f7080197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 -- 172.21.15.26:0/3252202344 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70740095f0 con 0x7f70800ff970 2024-03-24T10:40:43.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f70852c2700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.335+0000 7f7085ac3700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7070007e90 tx=0x7f707000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f707000de60 con 0x7f70800ff970 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f707000f040 con 0x7f70800ff970 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70801a0860 con 0x7f70800ff970 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70700045b0 con 0x7f70800ff970 2024-03-24T10:40:43.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70801a0cd0 con 0x7f70800ff970 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.336+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7080107f90 con 0x7f70800ff970 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.337+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f7070004ad0 con 0x7f70800ff970 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.337+0000 7f707effd700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 0x7f706806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.338+0000 7f70852c2700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 0x7f706806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.338+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f707008ba70 con 0x7f70800ff970 2024-03-24T10:40:43.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.338+0000 7f70852c2700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 0x7f706806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7080069560 tx=0x7f707401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.342+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7070056180 con 0x7f70800ff970 2024-03-24T10:40:43.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.587+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70801010b0 con 0x7f70800ff970 2024-03-24T10:40:43.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.588+0000 7f707effd700 1 -- 172.21.15.26:0/3252202344 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f70700597a0 con 0x7f70800ff970 2024-03-24T10:40:43.588 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:40:43.588 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 msgr2=0x7f706806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 0x7f706806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7080069560 tx=0x7f707401a040 comp rx=0 tx=0).stop 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 msgr2=0x7f70801977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7070007e90 tx=0x7f707000d3b0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 shutdown_connections 2024-03-24T10:40:43.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f706806cce0 0x7f706806f1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70800ff970 0x7f70801977f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 --2- 172.21.15.26:0/3252202344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080105970 0x7f7080197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 >> 172.21.15.26:0/3252202344 conn(0x7f70800fb480 msgr2=0x7f70800fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 shutdown_connections 2024-03-24T10:40:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:43.590+0000 7f7087d27700 1 -- 172.21.15.26:0/3252202344 wait complete. 2024-03-24T10:40:43.850 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:43 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3252202344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T10:40:43.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:43 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3252202344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T10:40:44.169 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T10:40:44.169 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T10:40:44.169 INFO:teuthology.run_tasks:Running task print... 2024-03-24T10:40:44.178 INFO:teuthology.task.print:**** done end installing reef cephadm ... 2024-03-24T10:40:44.178 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:40:44.267 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:40:44.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-24T10:40:44.480 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:44.925 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:44 smithi026 ceph-mon[29210]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:44.925 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:44 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:44.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:44 smithi031 ceph-mon[37345]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:44.950 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:44 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:46.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 -- 172.21.15.26:0/118727284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4101560 msgr2=0x7efdc4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 --2- 172.21.15.26:0/118727284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4101560 0x7efdc4103950 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efdb400b0a0 tx=0x7efdb400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 -- 172.21.15.26:0/118727284 shutdown_connections 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 --2- 172.21.15.26:0/118727284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4103e90 0x7efdc4106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 --2- 172.21.15.26:0/118727284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4101560 0x7efdc4103950 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 -- 172.21.15.26:0/118727284 >> 172.21.15.26:0/118727284 conn(0x7efdc40faf70 msgr2=0x7efdc40fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 -- 172.21.15.26:0/118727284 shutdown_connections 2024-03-24T10:40:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.066+0000 7efdcb335700 1 -- 172.21.15.26:0/118727284 wait complete. 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 Processor -- start 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 -- start start 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 0x7efdc41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4197030 con 0x7efdc4103e90 2024-03-24T10:40:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdcb335700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4197170 con 0x7efdc4101560 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59434/0 (socket says 172.21.15.26:59434) 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc90d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 0x7efdc41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 -- 172.21.15.26:0/4092087262 learned_addr learned my addr 172.21.15.26:0/4092087262 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 -- 172.21.15.26:0/4092087262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 msgr2=0x7efdc41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 0x7efdc41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 -- 172.21.15.26:0/4092087262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb4009d00 con 0x7efdc4103e90 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc90d1700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 0x7efdc41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.067+0000 7efdc88d0700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d900 tx=0x7efdb800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.068+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb80041f0 con 0x7efdc4103e90 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.068+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdb8004cf0 con 0x7efdc4103e90 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.068+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc419bbd0 con 0x7efdc4103e90 2024-03-24T10:40:46.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.068+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb800b750 con 0x7efdc4103e90 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.068+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc419c120 con 0x7efdc4103e90 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.069+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc404ea80 con 0x7efdc4103e90 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.069+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7efdb8004750 con 0x7efdc4103e90 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.069+0000 7efdc27fc700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 0x7efdac06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.069+0000 7efdc90d1700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 0x7efdac06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.069+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7efdb808bd50 con 0x7efdc4103e90 2024-03-24T10:40:46.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.070+0000 7efdc90d1700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 0x7efdac06f1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdb4000f80 tx=0x7efdb4007910 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:46.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.074+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efdb8059f70 con 0x7efdc4103e90 2024-03-24T10:40:46.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.280+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7efdc419c360 con 0x7efdc4103e90 2024-03-24T10:40:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.284+0000 7efdc27fc700 1 -- 172.21.15.26:0/4092087262 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7efdb80160d0 con 0x7efdc4103e90 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 msgr2=0x7efdac06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 0x7efdac06f1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdb4000f80 tx=0x7efdb4007910 comp rx=0 tx=0).stop 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 msgr2=0x7efdc4196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d900 tx=0x7efdb800dc10 comp rx=0 tx=0).stop 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 shutdown_connections 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efdac06ccf0 0x7efdac06f1b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc4101560 0x7efdc41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 --2- 172.21.15.26:0/4092087262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efdc4103e90 0x7efdc4196a10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 >> 172.21.15.26:0/4092087262 conn(0x7efdc40faf70 msgr2=0x7efdc40fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 shutdown_connections 2024-03-24T10:40:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:46.286+0000 7efdcb335700 1 -- 172.21.15.26:0/4092087262 wait complete. 2024-03-24T10:40:46.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:46 smithi031 ceph-mon[37345]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:46.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:46 smithi026 ceph-mon[29210]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:46.832 INFO:teuthology.run_tasks:Running task print... 2024-03-24T10:40:46.843 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-24T10:40:46.843 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:40:46.942 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:40:46.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T10:40:47.151 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:47.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:47 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/4092087262' entity='client.admin' 2024-03-24T10:40:47.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:47.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:47.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:47.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:47 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:40:47.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:47 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/4092087262' entity='client.admin' 2024-03-24T10:40:47.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:47.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:47.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:47.529 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:47 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:40:48.567 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:48 smithi026 ceph-mon[29210]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:48.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:48 smithi031 ceph-mon[37345]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 -- 172.21.15.26:0/3402882092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd34100bd0 msgr2=0x7efd34101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/3402882092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd34100bd0 0x7efd34101030 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efd28009910 tx=0x7efd28009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 -- 172.21.15.26:0/3402882092 shutdown_connections 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/3402882092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd34100bd0 0x7efd34101030 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/3402882092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd340ff9d0 0x7efd340ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 -- 172.21.15.26:0/3402882092 >> 172.21.15.26:0/3402882092 conn(0x7efd340faf70 msgr2=0x7efd340fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 -- 172.21.15.26:0/3402882092 shutdown_connections 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.812+0000 7efd3b3f4700 1 -- 172.21.15.26:0/3402882092 wait complete. 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 Processor -- start 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 -- start start 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd34100bd0 0x7efd34196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd341970e0 con 0x7efd340ff9d0 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3b3f4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd34197220 con 0x7efd34100bd0 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59450/0 (socket says 172.21.15.26:59450) 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 -- 172.21.15.26:0/864736919 learned_addr learned my addr 172.21.15.26:0/864736919 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 -- 172.21.15.26:0/864736919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd34100bd0 msgr2=0x7efd34196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd3898f700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd34100bd0 0x7efd34196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:48.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd34100bd0 0x7efd34196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 -- 172.21.15.26:0/864736919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280095f0 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.813+0000 7efd39190700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00d900 tx=0x7efd2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.814+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd2c0041f0 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.814+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd2c004350 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.814+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3419bc30 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.814+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd2c003dc0 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.814+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3419c180 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.815+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3404ea80 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.815+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7efd2c009730 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.816+0000 7efd327fc700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 0x7efd1c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.816+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7efd2c08c060 con 0x7efd340ff9d0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.816+0000 7efd3898f700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 0x7efd1c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.816+0000 7efd3898f700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 0x7efd1c06f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efd2800b5c0 tx=0x7efd28005160 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:48.820+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efd2c05a280 con 0x7efd340ff9d0 2024-03-24T10:40:49.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.037+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd340fcf20 con 0x7efd1c06cce0 2024-03-24T10:40:49.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.038+0000 7efd327fc700 1 -- 172.21.15.26:0/864736919 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7efd340fcf20 con 0x7efd1c06cce0 2024-03-24T10:40:49.039 INFO:teuthology.orchestra.run.smithi026.stdout:Backend: cephadm 2024-03-24T10:40:49.039 INFO:teuthology.orchestra.run.smithi026.stdout:Available: Yes 2024-03-24T10:40:49.039 INFO:teuthology.orchestra.run.smithi026.stdout:Paused: No 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 msgr2=0x7efd1c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 0x7efd1c06f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efd2800b5c0 tx=0x7efd28005160 comp rx=0 tx=0).stop 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 msgr2=0x7efd34196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00d900 tx=0x7efd2c00dcc0 comp rx=0 tx=0).stop 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 shutdown_connections 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7efd1c06cce0 0x7efd1c06f1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd340ff9d0 0x7efd34196580 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:49.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 --2- 172.21.15.26:0/864736919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd34100bd0 0x7efd34196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:49.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.040+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 >> 172.21.15.26:0/864736919 conn(0x7efd340faf70 msgr2=0x7efd340fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:49.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.041+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 shutdown_connections 2024-03-24T10:40:49.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:49.041+0000 7efd3b3f4700 1 -- 172.21.15.26:0/864736919 wait complete. 2024-03-24T10:40:49.673 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T10:40:49.887 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:49.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:49 smithi026 ceph-mon[29210]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:49.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:49 smithi026 ceph-mon[29210]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:50.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:49 smithi031 ceph-mon[37345]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:50.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:49 smithi031 ceph-mon[37345]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:51.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2489486582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 msgr2=0x7fdd6c0fefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:51.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2489486582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c0fefb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009910 tx=0x7fdd60009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2489486582 shutdown_connections 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2489486582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c0fefb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2489486582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2489486582 >> 172.21.15.26:0/2489486582 conn(0x7fdd6c0fa730 msgr2=0x7fdd6c0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2489486582 shutdown_connections 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.428+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2489486582 wait complete. 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 Processor -- start 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 -- start start 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c1970c0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd71bf8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c197200 con 0x7fdd6c1020d0 2024-03-24T10:40:51.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd6b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd6b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59470/0 (socket says 172.21.15.26:59470) 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.429+0000 7fdd6b7fe700 1 -- 172.21.15.26:0/2821466025 learned_addr learned my addr 172.21.15.26:0/2821466025 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6affd700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6b7fe700 1 -- 172.21.15.26:0/2821466025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 msgr2=0x7fdd6c196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6b7fe700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6b7fe700 1 -- 172.21.15.26:0/2821466025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600095f0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6affd700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd6b7fe700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00e8d0 tx=0x7fdd5c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd5c009770 con 0x7fdd6c0feb50 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c19bc60 con 0x7fdd6c0feb50 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd5c00eed0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd5c003e00 con 0x7fdd6c0feb50 2024-03-24T10:40:51.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.430+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c19c1b0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.431+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c1906a0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.431+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fdd5c00cba0 con 0x7fdd6c0feb50 2024-03-24T10:40:51.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.432+0000 7fdd68ff9700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 0x7fdd5406f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:51.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.432+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdd5c014070 con 0x7fdd6c0feb50 2024-03-24T10:40:51.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.432+0000 7fdd6affd700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 0x7fdd5406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:51.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.432+0000 7fdd6affd700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 0x7fdd5406f140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd600059d0 tx=0x7fdd60005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:51.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.436+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd5c05a490 con 0x7fdd6c0feb50 2024-03-24T10:40:51.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.647+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd6c0611f0 con 0x7fdd5406cc80 2024-03-24T10:40:51.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.656+0000 7fdd68ff9700 1 -- 172.21.15.26:0/2821466025 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3080 (secure 0 0 0) 0x7fdd6c0611f0 con 0x7fdd5406cc80 2024-03-24T10:40:51.656 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (3m) 2m ago 5m 26.6M - 0.25.0 c8568f914cd2 3f177e8f816c 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-exporter.smithi026 smithi026 running (5m) 2m ago 5m 7046k - 18.2.2-952-g738c2fd6 68aff9aff3b2 044d4f7bec30 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-exporter.smithi031 smithi031 running (4m) 68s ago 4m 6971k - 18.2.2-952-g738c2fd6 68aff9aff3b2 08932fd65525 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (5m) 2m ago 5m 6660k - 18.2.2-952-g738c2fd6 68aff9aff3b2 04efe48df2e6 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi031 smithi031 running (4m) 68s ago 4m 6660k - 18.2.2-952-g738c2fd6 68aff9aff3b2 c6b079e4b35c 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (3m) 2m ago 4m 78.2M - 9.4.7 954c08fa6188 4696b23e1eeb 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.bbjcru smithi026 *:9283,8765,8443 running (6m) 2m ago 6m 489M - 18.2.2-952-g738c2fd6 68aff9aff3b2 181e5b6ed806 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi031.ucwncv smithi031 *:8443,9283,8765 running (4m) 68s ago 4m 450M - 18.2.2-952-g738c2fd6 68aff9aff3b2 ff3fb3d02a0e 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (6m) 2m ago 6m 46.1M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 dfd13627777b 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi031 smithi031 running (4m) 68s ago 4m 40.0M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 72bc55976cf6 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (5m) 2m ago 5m 16.0M - 1.5.0 0da6a335fe13 850149efce92 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi031 smithi031 *:9100 running (4m) 68s ago 4m 15.3M - 1.5.0 0da6a335fe13 ef178edadca0 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (3m) 2m ago 3m 42.4M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 6dd317107551 2024-03-24T10:40:51.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (2m) 2m ago 2m 39.7M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 8bf2a3e7910d 2024-03-24T10:40:51.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (2m) 2m ago 2m 39.9M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 d9c7f3ffbeae 2024-03-24T10:40:51.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi031 running (2m) 68s ago 2m 40.4M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 afb16aab0936 2024-03-24T10:40:51.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi031 running (98s) 68s ago 98s 41.3M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 6b4b79fce473 2024-03-24T10:40:51.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi031 running (73s) 68s ago 73s 13.6M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 1851387fda3d 2024-03-24T10:40:51.658 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (3m) 2m ago 4m 39.8M - 2.43.0 a07b618ecd1d fb76b5ad356d 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 msgr2=0x7fdd5406f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 0x7fdd5406f140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd600059d0 tx=0x7fdd60005940 comp rx=0 tx=0).stop 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 msgr2=0x7fdd6c196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00e8d0 tx=0x7fdd5c00ec90 comp rx=0 tx=0).stop 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 shutdown_connections 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fdd5406cc80 0x7fdd5406f140 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd6c0feb50 0x7fdd6c196560 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 --2- 172.21.15.26:0/2821466025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd6c1020d0 0x7fdd6c196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 >> 172.21.15.26:0/2821466025 conn(0x7fdd6c0fa730 msgr2=0x7fdd6c105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 shutdown_connections 2024-03-24T10:40:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:51.658+0000 7fdd71bf8700 1 -- 172.21.15.26:0/2821466025 wait complete. 2024-03-24T10:40:51.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:51 smithi026 ceph-mon[29210]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:51.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:51 smithi026 ceph-mon[29210]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:51 smithi031 ceph-mon[37345]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:51 smithi031 ceph-mon[37345]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:52.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T10:40:52.616 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:53.851 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:53 smithi026 ceph-mon[29210]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- 172.21.15.26:0/2285485565 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 msgr2=0x7f7064100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 --2- 172.21.15.26:0/2285485565 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064100f20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f705c00b0a0 tx=0x7f705c00b3b0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- 172.21.15.26:0/2285485565 shutdown_connections 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 --2- 172.21.15.26:0/2285485565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 0x7f7064102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 --2- 172.21.15.26:0/2285485565 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064100f20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- 172.21.15.26:0/2285485565 >> 172.21.15.26:0/2285485565 conn(0x7f70640fc0a0 msgr2=0x7f70640fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- 172.21.15.26:0/2285485565 shutdown_connections 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- 172.21.15.26:0/2285485565 wait complete. 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 Processor -- start 2024-03-24T10:40:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.140+0000 7f706956b700 1 -- start start 2024-03-24T10:40:54.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f706956b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:54.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f706956b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 0x7f70640731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:54.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f706956b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640736e0 con 0x7f7064100b00 2024-03-24T10:40:54.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f706956b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064073820 con 0x7f7064101d00 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50242/0 (socket says 172.21.15.26:50242) 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 -- 172.21.15.26:0/3862736443 learned_addr learned my addr 172.21.15.26:0/3862736443 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 -- 172.21.15.26:0/3862736443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 msgr2=0x7f70640731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f70627fc700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 0x7f70640731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 0x7f70640731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 -- 172.21.15.26:0/3862736443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c009d00 con 0x7f7064100b00 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f7062ffd700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f705c015040 tx=0x7f705c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c021070 con 0x7f7064100b00 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7064073aa0 con 0x7f7064100b00 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f705c0075c0 con 0x7f7064100b00 2024-03-24T10:40:54.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.141+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c00e040 con 0x7f7064100b00 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.142+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7064074010 con 0x7f7064100b00 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.143+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064190700 con 0x7f7064100b00 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.143+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f705c004250 con 0x7f7064100b00 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.143+0000 7f705bfff700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 0x7f705006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.144+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f705c08cfe0 con 0x7f7064100b00 2024-03-24T10:40:54.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.144+0000 7f70627fc700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 0x7f705006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:54.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.144+0000 7f70627fc700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 0x7f705006f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f704c006fd0 tx=0x7f704c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.148+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f705c05b180 con 0x7f7064100b00 2024-03-24T10:40:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:53 smithi031 ceph-mon[37345]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:54.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.361+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7064074010 con 0x7f705006cce0 2024-03-24T10:40:54.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.366+0000 7f705bfff700 1 -- 172.21.15.26:0/3862736443 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1290 (secure 0 0 0) 0x7f7064074010 con 0x7f705006cce0 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-exporter 2/2 2m ago 6m * 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:crash 2/2 2m ago 6m * 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:mgr 2/2 2m ago 6m count:2 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:mon 2/2 2m ago 5m smithi026:172.21.15.26=smithi026;smithi031:172.21.15.31=smithi031;count:2 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd 6 2m ago - 2024-03-24T10:40:54.367 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 msgr2=0x7f705006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 0x7f705006f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f704c006fd0 tx=0x7f704c008040 comp rx=0 tx=0).stop 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 msgr2=0x7f7064074ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f705c015040 tx=0x7f705c003ce0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 shutdown_connections 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f705006cce0 0x7f705006f1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064100b00 0x7f7064074ab0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 --2- 172.21.15.26:0/3862736443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7064101d00 0x7f70640731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 >> 172.21.15.26:0/3862736443 conn(0x7f70640fc0a0 msgr2=0x7f7064105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 shutdown_connections 2024-03-24T10:40:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:54.368+0000 7f706956b700 1 -- 172.21.15.26:0/3862736443 wait complete. 2024-03-24T10:40:54.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:54 smithi026 ceph-mon[29210]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:54.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T10:40:55.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:54 smithi031 ceph-mon[37345]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:56.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:56 smithi031 ceph-mon[37345]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:56.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:56 smithi026 ceph-mon[29210]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 -- 172.21.15.26:0/1740775968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554100ae0 msgr2=0x7f0554100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1740775968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554100ae0 0x7f0554100f00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0550009910 tx=0x7f0550009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 -- 172.21.15.26:0/1740775968 shutdown_connections 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1740775968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554101ce0 0x7f0554102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1740775968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554100ae0 0x7f0554100f00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:56.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 -- 172.21.15.26:0/1740775968 >> 172.21.15.26:0/1740775968 conn(0x7f05540fc0a0 msgr2=0x7f05540fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 -- 172.21.15.26:0/1740775968 shutdown_connections 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.804+0000 7f055c0fb700 1 -- 172.21.15.26:0/1740775968 wait complete. 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 Processor -- start 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 -- start start 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554100ae0 0x7f05541964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554197030 con 0x7f0554101ce0 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f055c0fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554197170 con 0x7f0554100ae0 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50262/0 (socket says 172.21.15.26:50262) 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 -- 172.21.15.26:0/1318463975 learned_addr learned my addr 172.21.15.26:0/1318463975 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559e97700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554100ae0 0x7f05541964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 -- 172.21.15.26:0/1318463975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554100ae0 msgr2=0x7f05541964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:56.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554100ae0 0x7f05541964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.805+0000 7f0559696700 1 -- 172.21.15.26:0/1318463975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500095f0 con 0x7f0554101ce0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f0559696700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f054400c8a0 tx=0x7f054400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054400cea0 con 0x7f0554101ce0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055419bbd0 con 0x7f0554101ce0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0544004830 con 0x7f0554101ce0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05440055b0 con 0x7f0554101ce0 2024-03-24T10:40:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.806+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055419c1a0 con 0x7f0554101ce0 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.807+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0554190660 con 0x7f0554101ce0 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.807+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f05440049a0 con 0x7f0554101ce0 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.808+0000 7f054affd700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 0x7f054006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.808+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f054408bba0 con 0x7f0554101ce0 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.808+0000 7f0559e97700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 0x7f054006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.808+0000 7f0559e97700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 0x7f054006f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0550006010 tx=0x7f055000b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:56.812+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0544059d40 con 0x7f0554101ce0 2024-03-24T10:40:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.022+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f05540611f0 con 0x7f054006cce0 2024-03-24T10:40:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.024+0000 7f054affd700 1 -- 172.21.15.26:0/1318463975 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f05540611f0 con 0x7f054006cce0 2024-03-24T10:40:57.025 INFO:teuthology.orchestra.run.smithi026.stdout:HOST ADDR LABELS STATUS 2024-03-24T10:40:57.025 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 172.21.15.26 2024-03-24T10:40:57.026 INFO:teuthology.orchestra.run.smithi026.stdout:smithi031 172.21.15.31 2024-03-24T10:40:57.026 INFO:teuthology.orchestra.run.smithi026.stdout:2 hosts in cluster 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 msgr2=0x7f054006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 0x7f054006f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0550006010 tx=0x7f055000b580 comp rx=0 tx=0).stop 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 msgr2=0x7f0554196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f054400c8a0 tx=0x7f054400cc60 comp rx=0 tx=0).stop 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 shutdown_connections 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f054006cce0 0x7f054006f1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0554100ae0 0x7f05541964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 --2- 172.21.15.26:0/1318463975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0554101ce0 0x7f0554196a10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 >> 172.21.15.26:0/1318463975 conn(0x7f05540fc0a0 msgr2=0x7f05541052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 shutdown_connections 2024-03-24T10:40:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:57.026+0000 7f055c0fb700 1 -- 172.21.15.26:0/1318463975 wait complete. 2024-03-24T10:40:57.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T10:40:57.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:57 smithi026 ceph-mon[29210]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:57.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:57 smithi026 ceph-mon[29210]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:57.974 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:40:58.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:57 smithi031 ceph-mon[37345]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:58.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:57 smithi031 ceph-mon[37345]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:40:59.019 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:59.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:40:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.509+0000 7f64acd9a700 1 -- 172.21.15.26:0/3568187929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 msgr2=0x7f64a80ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.509+0000 7f64acd9a700 1 --2- 172.21.15.26:0/3568187929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a80ffe00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f649c009910 tx=0x7f649c009c20 comp rx=0 tx=0).stop 2024-03-24T10:40:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- 172.21.15.26:0/3568187929 shutdown_connections 2024-03-24T10:40:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 --2- 172.21.15.26:0/3568187929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 0x7f64a8101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 --2- 172.21.15.26:0/3568187929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a80ffe00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- 172.21.15.26:0/3568187929 >> 172.21.15.26:0/3568187929 conn(0x7f64a80faf80 msgr2=0x7f64a80fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- 172.21.15.26:0/3568187929 shutdown_connections 2024-03-24T10:40:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- 172.21.15.26:0/3568187929 wait complete. 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 Processor -- start 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- start start 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 0x7f64a8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8197050 con 0x7f64a80ff9e0 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.510+0000 7f64acd9a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8197190 con 0x7f64a8100be0 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50270/0 (socket says 172.21.15.26:50270) 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 -- 172.21.15.26:0/2648411493 learned_addr learned my addr 172.21.15.26:0/2648411493 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a5d9b700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 0x7f64a8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 -- 172.21.15.26:0/2648411493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 msgr2=0x7f64a8196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 0x7f64a8196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 -- 172.21.15.26:0/2648411493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0095f0 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64a659c700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f649c0059d0 tx=0x7f649c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649c01d070 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f649c005dc0 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a819bb90 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649c021410 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.511+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a819c080 con 0x7f64a80ff9e0 2024-03-24T10:40:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.512+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a804ea80 con 0x7f64a80ff9e0 2024-03-24T10:40:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.513+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f649c00f460 con 0x7f64a80ff9e0 2024-03-24T10:40:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.513+0000 7f64977fe700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 0x7f649006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.513+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f649c08d110 con 0x7f64a80ff9e0 2024-03-24T10:40:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.513+0000 7f64a5d9b700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 0x7f649006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.514+0000 7f64a5d9b700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 0x7f649006f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6498005950 tx=0x7f649800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.518+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f649c05b2b0 con 0x7f64a80ff9e0 2024-03-24T10:40:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.730+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f64a80697b0 con 0x7f649006cce0 2024-03-24T10:40:59.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.732+0000 7f64977fe700 1 -- 172.21.15.26:0/2648411493 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f64a80697b0 con 0x7f649006cce0 2024-03-24T10:40:59.733 INFO:teuthology.orchestra.run.smithi026.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T10:40:59.733 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760020480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-24T10:40:59.734 INFO:teuthology.orchestra.run.smithi026.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 72s ago Has a FileSystem, LVM detected 2024-03-24T10:40:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 msgr2=0x7f649006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 0x7f649006f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6498005950 tx=0x7f649800b410 comp rx=0 tx=0).stop 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 msgr2=0x7f64a81964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f649c0059d0 tx=0x7f649c00bd70 comp rx=0 tx=0).stop 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 shutdown_connections 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f649006cce0 0x7f649006f1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64a80ff9e0 0x7f64a81964f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 --2- 172.21.15.26:0/2648411493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a8100be0 0x7f64a8196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 >> 172.21.15.26:0/2648411493 conn(0x7f64a80faf80 msgr2=0x7f64a80690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 shutdown_connections 2024-03-24T10:40:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:40:59.734+0000 7f64acd9a700 1 -- 172.21.15.26:0/2648411493 wait complete. 2024-03-24T10:41:00.042 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:40:59 smithi026 ceph-mon[29210]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:00.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:40:59 smithi031 ceph-mon[37345]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:00.306 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:41:00.315 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:00.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-24T10:41:00.523 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:01 smithi026 ceph-mon[29210]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:01.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:01 smithi031 ceph-mon[37345]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:02.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 -- 172.21.15.26:0/807533369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 msgr2=0x7f74d4103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:02.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/807533369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d4103990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009910 tx=0x7f74bc009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 -- 172.21.15.26:0/807533369 shutdown_connections 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/807533369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d41062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/807533369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d4103990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 -- 172.21.15.26:0/807533369 >> 172.21.15.26:0/807533369 conn(0x7f74d40fafb0 msgr2=0x7f74d40fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 -- 172.21.15.26:0/807533369 shutdown_connections 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.076+0000 7f74d9dd1700 1 -- 172.21.15.26:0/807533369 wait complete. 2024-03-24T10:41:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.077+0000 7f74d9dd1700 1 Processor -- start 2024-03-24T10:41:02.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.077+0000 7f74d9dd1700 1 -- start start 2024-03-24T10:41:02.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d4196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4197140 con 0x7f74d41015a0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4197280 con 0x7f74d4103ed0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50276/0 (socket says 172.21.15.26:50276) 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 -- 172.21.15.26:0/4198921130 learned_addr learned my addr 172.21.15.26:0/4198921130 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d2ffd700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d4196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 -- 172.21.15.26:0/4198921130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 msgr2=0x7f74d4196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d4196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 -- 172.21.15.26:0/4198921130 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc0095f0 con 0x7f74d41015a0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d2ffd700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d4196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d37fe700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00b5c0 tx=0x7f74bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc01d070 con 0x7f74d41015a0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74bc00f460 con 0x7f74d41015a0 2024-03-24T10:41:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d419bc80 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc0175f0 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.078+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d419c170 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.079+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d404ea80 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.080+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f74bc00f5d0 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.080+0000 7f74d0ff9700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 0x7f74c006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.080+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f74bc08d4e0 con 0x7f74d41015a0 2024-03-24T10:41:02.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.080+0000 7f74d2ffd700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 0x7f74c006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:02.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.081+0000 7f74d2ffd700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 0x7f74c006f1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f74c40099d0 tx=0x7f74c4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:02.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.085+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f74bc05b680 con 0x7f74d41015a0 2024-03-24T10:41:02.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:02.316+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f74d4104910 con 0x7f74c006ccf0 2024-03-24T10:41:02.449 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:02 smithi031 ceph-mon[37345]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:02.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:02 smithi026 ceph-mon[29210]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:03 smithi031 ceph-mon[37345]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:03.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:03 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T10:41:03.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:03 smithi026 ceph-mon[29210]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:03.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:03 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T10:41:04.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.207+0000 7f74d0ff9700 1 -- 172.21.15.26:0/4198921130 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f74d4104910 con 0x7f74c006ccf0 2024-03-24T10:41:04.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 msgr2=0x7f74c006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:04.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 0x7f74c006f1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f74c40099d0 tx=0x7f74c4008040 comp rx=0 tx=0).stop 2024-03-24T10:41:04.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 msgr2=0x7f74d41965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:04.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00b5c0 tx=0x7f74bc00be20 comp rx=0 tx=0).stop 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 shutdown_connections 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74c006ccf0 0x7f74c006f1b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74d41015a0 0x7f74d41965e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 --2- 172.21.15.26:0/4198921130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74d4103ed0 0x7f74d4196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 >> 172.21.15.26:0/4198921130 conn(0x7f74d40fafb0 msgr2=0x7f74d40fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 shutdown_connections 2024-03-24T10:41:04.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:04.211+0000 7f74d9dd1700 1 -- 172.21.15.26:0/4198921130 wait complete. 2024-03-24T10:41:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:04 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T10:41:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:04 smithi031 ceph-mon[37345]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T10:41:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:04 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T10:41:04.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:04 smithi031 ceph-mon[37345]: pgmap v175: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:04 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T10:41:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:04 smithi026 ceph-mon[29210]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T10:41:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:04 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T10:41:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:04 smithi026 ceph-mon[29210]: pgmap v175: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:04.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:41:04.184+0000 7f9761826700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:41:04.919 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T10:41:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:05.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:41:05.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T10:41:05.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T10:41:05.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T10:41:05.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: fsmap cephfs:0 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: Saving service mds.cephfs spec with placement count:4 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.dfxoad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.dfxoad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:05.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:05 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: fsmap cephfs:0 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: Saving service mds.cephfs spec with placement count:4 2024-03-24T10:41:05.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.dfxoad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.dfxoad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:05.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:05 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:06.372 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:06 smithi026 ceph-mon[29210]: Deploying daemon mds.cephfs.smithi026.dfxoad on smithi026 2024-03-24T10:41:06.372 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:06 smithi026 ceph-mon[29210]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T10:41:06.372 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:06 smithi026 ceph-mon[29210]: pgmap v179: 65 pgs: 25 active+clean, 13 creating+peering, 27 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:06 smithi031 ceph-mon[37345]: Deploying daemon mds.cephfs.smithi026.dfxoad on smithi026 2024-03-24T10:41:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:06 smithi031 ceph-mon[37345]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T10:41:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:06 smithi031 ceph-mon[37345]: pgmap v179: 65 pgs: 25 active+clean, 13 creating+peering, 27 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 -- 172.21.15.26:0/3421858346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93301015a0 msgr2=0x7f9330103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 --2- 172.21.15.26:0/3421858346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93301015a0 0x7f9330103990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f932000b0a0 tx=0x7f932000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 -- 172.21.15.26:0/3421858346 shutdown_connections 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 --2- 172.21.15.26:0/3421858346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9330103ed0 0x7f93301062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 --2- 172.21.15.26:0/3421858346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93301015a0 0x7f9330103990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 -- 172.21.15.26:0/3421858346 >> 172.21.15.26:0/3421858346 conn(0x7f93300fafb0 msgr2=0x7f93300fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 -- 172.21.15.26:0/3421858346 shutdown_connections 2024-03-24T10:41:06.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.716+0000 7f9336b85700 1 -- 172.21.15.26:0/3421858346 wait complete. 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 Processor -- start 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 -- start start 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93301015a0 0x7f93301964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330197030 con 0x7f9330103ed0 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f9336b85700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330197170 con 0x7f93301015a0 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f932ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.717+0000 7f932ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60102/0 (socket says 172.21.15.26:60102) 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932ffff700 1 -- 172.21.15.26:0/1601714078 learned_addr learned my addr 172.21.15.26:0/1601714078 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f9334921700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93301015a0 0x7f93301964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932ffff700 1 -- 172.21.15.26:0/1601714078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93301015a0 msgr2=0x7f93301964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932ffff700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93301015a0 0x7f93301964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932ffff700 1 -- 172.21.15.26:0/1601714078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9320009d00 con 0x7f9330103ed0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932ffff700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f932400c8a0 tx=0x7f932400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93240078e0 con 0x7f9330103ed0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933019bbd0 con 0x7f9330103ed0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f932400f450 con 0x7f9330103ed0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f932400e5e0 con 0x7f9330103ed0 2024-03-24T10:41:06.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.718+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933019c1a0 con 0x7f9330103ed0 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.719+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933004ea80 con 0x7f9330103ed0 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.720+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f932400f5c0 con 0x7f9330103ed0 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.720+0000 7f932dffb700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 0x7f931806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.720+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f932408c110 con 0x7f9330103ed0 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.721+0000 7f9334921700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 0x7f931806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.721+0000 7f9334921700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 0x7f931806f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9320000f80 tx=0x7f9320006000 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.725+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9324059ff0 con 0x7f9330103ed0 2024-03-24T10:41:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.977+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f933019c450 con 0x7f9330103ed0 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.977+0000 7f932dffb700 1 -- 172.21.15.26:0/1601714078 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1114 (secure 0 0 0) 0x7f9324059b80 con 0x7f9330103ed0 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:e2 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-03-24T10:41:06.978 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 2 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-03-24T10:41:04.185237+0000 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-03-24T10:41:04.185306+0000 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 0 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:41:06.979 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:in 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:up {} 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 0 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 msgr2=0x7f931806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 0x7f931806f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9320000f80 tx=0x7f9320006000 comp rx=0 tx=0).stop 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 msgr2=0x7f9330196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f932400c8a0 tx=0x7f932400cbb0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 shutdown_connections 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f931806cce0 0x7f931806f1a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93301015a0 0x7f93301964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 --2- 172.21.15.26:0/1601714078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9330103ed0 0x7f9330196a10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 >> 172.21.15.26:0/1601714078 conn(0x7f93300fafb0 msgr2=0x7f93300fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 shutdown_connections 2024-03-24T10:41:06.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:06.980+0000 7f9336b85700 1 -- 172.21.15.26:0/1601714078 wait complete. 2024-03-24T10:41:06.982 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 2 2024-03-24T10:41:07.272 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:07 smithi026 ceph-mon[29210]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T10:41:07.273 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:07 smithi026 ceph-mon[29210]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T10:41:07.273 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:07 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1601714078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:41:07.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:07 smithi031 ceph-mon[37345]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T10:41:07.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:07 smithi031 ceph-mon[37345]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T10:41:07.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:07 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1601714078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:41:07.920 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:41:07.929 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:07.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-03-24T10:41:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:08.475 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:08 smithi026 ceph-mon[29210]: pgmap v181: 65 pgs: 47 active+clean, 13 creating+peering, 5 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:08.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:08 smithi031 ceph-mon[37345]: pgmap v181: 65 pgs: 47 active+clean, 13 creating+peering, 5 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:09.843 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:09 smithi026 ceph-mon[29210]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:10.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:09 smithi031 ceph-mon[37345]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:41:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 -- 172.21.15.26:0/1441242450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 msgr2=0x7f3040106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 --2- 172.21.15.26:0/1441242450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f3040106280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3030009910 tx=0x7f3030009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 -- 172.21.15.26:0/1441242450 shutdown_connections 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 --2- 172.21.15.26:0/1441242450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f3040106280 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 --2- 172.21.15.26:0/1441242450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 0x7f3040103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 -- 172.21.15.26:0/1441242450 >> 172.21.15.26:0/1441242450 conn(0x7f30400faf70 msgr2=0x7f30400fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 -- 172.21.15.26:0/1441242450 shutdown_connections 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.647+0000 7f3047b5d700 1 -- 172.21.15.26:0/1441242450 wait complete. 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 Processor -- start 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 -- start start 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 0x7f3040100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400ffa50 con 0x7f3040103e90 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f3047b5d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400ffb90 con 0x7f3040101560 2024-03-24T10:41:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60138/0 (socket says 172.21.15.26:60138) 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 -- 172.21.15.26:0/557867159 learned_addr learned my addr 172.21.15.26:0/557867159 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30458f9700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 0x7f3040100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 -- 172.21.15.26:0/557867159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 msgr2=0x7f3040100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 0x7f3040100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 -- 172.21.15.26:0/557867159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300095f0 con 0x7f3040103e90 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.648+0000 7f30450f8700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3030006010 tx=0x7f303000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303001d070 con 0x7f3040103e90 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f303000f460 con 0x7f3040103e90 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30400ffe10 con 0x7f3040103e90 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3030021620 con 0x7f3040103e90 2024-03-24T10:41:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3040100300 con 0x7f3040103e90 2024-03-24T10:41:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.649+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304004ea80 con 0x7f3040103e90 2024-03-24T10:41:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.650+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f303000f5d0 con 0x7f3040103e90 2024-03-24T10:41:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.650+0000 7f3036ffd700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 0x7f302c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.651+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f303008d460 con 0x7f3040103e90 2024-03-24T10:41:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.651+0000 7f30458f9700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 0x7f302c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.651+0000 7f30458f9700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 0x7f302c06f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f303c007c60 tx=0x7f303c0073d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.655+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f303005b2c0 con 0x7f3040103e90 2024-03-24T10:41:10.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:10.908+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f304019e840 con 0x7f3040103e90 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: daemon mds.cephfs.smithi026.dfxoad assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: Cluster is now healthy 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6826/2645357503,v1:172.21.15.26:6827/2645357503] up:boot 2024-03-24T10:41:11.153 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:creating} 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: daemon mds.cephfs.smithi026.dfxoad is now active in filesystem cephfs as rank 0 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.cgovmq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.cgovmq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:11.154 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:10 smithi031 ceph-mon[37345]: Deploying daemon mds.cephfs.smithi031.cgovmq on smithi031 2024-03-24T10:41:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.168+0000 7f3036ffd700 1 -- 172.21.15.26:0/557867159 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v4) v1 ==== 105+0+0 (secure 0 0 0) 0x7f303005ae50 con 0x7f3040103e90 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: daemon mds.cephfs.smithi026.dfxoad assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: Cluster is now healthy 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6826/2645357503,v1:172.21.15.26:6827/2645357503] up:boot 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:creating} 2024-03-24T10:41:11.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: daemon mds.cephfs.smithi026.dfxoad is now active in filesystem cephfs as rank 0 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.cgovmq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.cgovmq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:11.170 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:10 smithi026 ceph-mon[29210]: Deploying daemon mds.cephfs.smithi031.cgovmq on smithi031 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 msgr2=0x7f302c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 0x7f302c06f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f303c007c60 tx=0x7f303c0073d0 comp rx=0 tx=0).stop 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 msgr2=0x7f30400ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3030006010 tx=0x7f303000bd70 comp rx=0 tx=0).stop 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 shutdown_connections 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f302c06cce0 0x7f302c06f1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3040101560 0x7f3040100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 --2- 172.21.15.26:0/557867159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3040103e90 0x7f30400ff510 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 >> 172.21.15.26:0/557867159 conn(0x7f30400faf70 msgr2=0x7f30400fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 shutdown_connections 2024-03-24T10:41:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:11.171+0000 7f3047b5d700 1 -- 172.21.15.26:0/557867159 wait complete. 2024-03-24T10:41:11.814 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:41:11.825 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:11.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-03-24T10:41:12.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:11 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/557867159' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T10:41:12.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:11 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6826/2645357503,v1:172.21.15.26:6827/2645357503] up:active 2024-03-24T10:41:12.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:11 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/557867159' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T10:41:12.005 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:11 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2024-03-24T10:41:12.006 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:11 smithi031 ceph-mon[37345]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s 2024-03-24T10:41:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:12.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:11 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/557867159' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T10:41:12.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:11 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6826/2645357503,v1:172.21.15.26:6827/2645357503] up:active 2024-03-24T10:41:12.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:11 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/557867159' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T10:41:12.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:11 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2024-03-24T10:41:12.169 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:11 smithi026 ceph-mon[29210]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 -- 172.21.15.26:0/2612466734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 msgr2=0x7f74c80ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/2612466734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c80ffe30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009910 tx=0x7f74b8009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 -- 172.21.15.26:0/2612466734 shutdown_connections 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/2612466734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 0x7f74c8101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/2612466734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c80ffe30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 -- 172.21.15.26:0/2612466734 >> 172.21.15.26:0/2612466734 conn(0x7f74c80fafb0 msgr2=0x7f74c80fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 -- 172.21.15.26:0/2612466734 shutdown_connections 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 -- 172.21.15.26:0/2612466734 wait complete. 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.651+0000 7f74ce1e7700 1 Processor -- start 2024-03-24T10:41:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 -- start start 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 0x7f74c8102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8102950 con 0x7f74c80ffa10 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8102a90 con 0x7f74c8100c10 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53916/0 (socket says 172.21.15.26:53916) 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 -- 172.21.15.26:0/3676111790 learned_addr learned my addr 172.21.15.26:0/3676111790 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c6ffd700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 0x7f74c8102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 -- 172.21.15.26:0/3676111790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 msgr2=0x7f74c8102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 0x7f74c8102410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 -- 172.21.15.26:0/3676111790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80095f0 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c77fe700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f74b8005850 tx=0x7f74b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b801d070 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c8102d10 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74b800f460 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b8017610 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.652+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8103280 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.654+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c804ea80 con 0x7f74c80ffa10 2024-03-24T10:41:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.654+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f74b800f5d0 con 0x7f74c80ffa10 2024-03-24T10:41:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.654+0000 7f74c4ff9700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 0x7f74b006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.655+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f74b808d690 con 0x7f74c80ffa10 2024-03-24T10:41:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.655+0000 7f74c6ffd700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 0x7f74b006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.655+0000 7f74c6ffd700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 0x7f74b006f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005fd0 tx=0x7f74bc00a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:13.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.659+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f74b805b4f0 con 0x7f74c80ffa10 2024-03-24T10:41:13.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:13.928+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f74c8103280 con 0x7f74c80ffa10 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.ryhyqm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.ryhyqm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:14.182 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: Deploying daemon mds.cephfs.smithi026.ryhyqm on smithi026 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.31:6824/3250075277,v1:172.21.15.31:6825/3250075277] up:boot 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi031.cgovmq"}]: dispatch 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 240 B/s wr, 2 op/s 2024-03-24T10:41:14.183 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.ryhyqm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi026.ryhyqm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: Deploying daemon mds.cephfs.smithi026.ryhyqm on smithi026 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.31:6824/3250075277,v1:172.21.15.31:6825/3250075277] up:boot 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi031.cgovmq"}]: dispatch 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 240 B/s wr, 2 op/s 2024-03-24T10:41:14.201 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:14.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.922+0000 7f74c4ff9700 1 -- 172.21.15.26:0/3676111790 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f74b805b080 con 0x7f74c80ffa10 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 msgr2=0x7f74b006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 0x7f74b006f1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005fd0 tx=0x7f74bc00a560 comp rx=0 tx=0).stop 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 msgr2=0x7f74c8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f74b8005850 tx=0x7f74b800be20 comp rx=0 tx=0).stop 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 shutdown_connections 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f74b006cce0 0x7f74b006f1a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:14.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74c80ffa10 0x7f74c8103d70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:14.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 --2- 172.21.15.26:0/3676111790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74c8100c10 0x7f74c8102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:14.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.925+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 >> 172.21.15.26:0/3676111790 conn(0x7f74c80fafb0 msgr2=0x7f74c8069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:14.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.926+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 shutdown_connections 2024-03-24T10:41:14.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:14.926+0000 7f74ce1e7700 1 -- 172.21.15.26:0/3676111790 wait complete. 2024-03-24T10:41:15.186 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:14 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3676111790' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-24T10:41:15.195 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:14 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3676111790' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-24T10:41:15.595 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:41:15.606 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:15.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-24T10:41:16.150 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:15 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3676111790' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-24T10:41:16.150 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:15 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:16.150 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:15 smithi031 ceph-mon[37345]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-24T10:41:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:15 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3676111790' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-24T10:41:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:15 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 1 up:standby 2024-03-24T10:41:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:15 smithi026 ceph-mon[29210]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-24T10:41:16.263 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.vjdtgj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.vjdtgj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: Deploying daemon mds.cephfs.smithi031.vjdtgj on smithi031 2024-03-24T10:41:17.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:boot 2024-03-24T10:41:17.418 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2 up:standby 2024-03-24T10:41:17.418 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:17 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.ryhyqm"}]: dispatch 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.vjdtgj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi031.vjdtgj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:41:17.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:17.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: Deploying daemon mds.cephfs.smithi031.vjdtgj on smithi031 2024-03-24T10:41:17.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:boot 2024-03-24T10:41:17.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2 up:standby 2024-03-24T10:41:17.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:17 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.ryhyqm"}]: dispatch 2024-03-24T10:41:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.781+0000 7f496effc700 1 -- 172.21.15.26:0/1620268634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 msgr2=0x7f49680ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.781+0000 7f496effc700 1 --2- 172.21.15.26:0/1620268634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f49680ffe30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f495c009910 tx=0x7f495c009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- 172.21.15.26:0/1620268634 shutdown_connections 2024-03-24T10:41:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 --2- 172.21.15.26:0/1620268634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 0x7f4968101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 --2- 172.21.15.26:0/1620268634 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f49680ffe30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- 172.21.15.26:0/1620268634 >> 172.21.15.26:0/1620268634 conn(0x7f49680faf70 msgr2=0x7f49680fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- 172.21.15.26:0/1620268634 shutdown_connections 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- 172.21.15.26:0/1620268634 wait complete. 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 Processor -- start 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- start start 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 0x7f4968195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968195fb0 con 0x7f49680ffa10 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496effc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49681960f0 con 0x7f4968100c10 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.782+0000 7f496cd98700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53956/0 (socket says 172.21.15.26:53956) 2024-03-24T10:41:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 -- 172.21.15.26:0/303502814 learned_addr learned my addr 172.21.15.26:0/303502814 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f4967fff700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 0x7f4968195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 -- 172.21.15.26:0/303502814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 msgr2=0x7f4968195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 0x7f4968195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 -- 172.21.15.26:0/303502814 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c0095f0 con 0x7f49680ffa10 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496cd98700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f495c0059d0 tx=0x7f495c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495c01d070 con 0x7f49680ffa10 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496819aaa0 con 0x7f49680ffa10 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f495c00f460 con 0x7f49680ffa10 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495c021620 con 0x7f49680ffa10 2024-03-24T10:41:17.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.783+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496819b010 con 0x7f49680ffa10 2024-03-24T10:41:17.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.784+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496818f590 con 0x7f49680ffa10 2024-03-24T10:41:17.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.785+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f495c00fa70 con 0x7f49680ffa10 2024-03-24T10:41:17.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.785+0000 7f4965ffb700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 0x7f495806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.785+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f495c08d5d0 con 0x7f49680ffa10 2024-03-24T10:41:17.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.785+0000 7f4967fff700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 0x7f495806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.786+0000 7f4967fff700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 0x7f495806f1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4954005fd0 tx=0x7f495400a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:17.790+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f495c05b430 con 0x7f49680ffa10 2024-03-24T10:41:18.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.045+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f496804ea80 con 0x7f49680ffa10 2024-03-24T10:41:18.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.151+0000 7f4965ffb700 1 -- 172.21.15.26:0/303502814 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v9) v1 ==== 168+0+0 (secure 0 0 0) 0x7f495c05afc0 con 0x7f49680ffa10 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 msgr2=0x7f495806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 0x7f495806f1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4954005fd0 tx=0x7f495400a560 comp rx=0 tx=0).stop 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 msgr2=0x7f4968195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f495c0059d0 tx=0x7f495c00be20 comp rx=0 tx=0).stop 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 shutdown_connections 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f495806cce0 0x7f495806f1a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49680ffa10 0x7f4968195450 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 --2- 172.21.15.26:0/303502814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4968100c10 0x7f4968195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 >> 172.21.15.26:0/303502814 conn(0x7f49680faf70 msgr2=0x7f49681041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 shutdown_connections 2024-03-24T10:41:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:18.153+0000 7f496effc700 1 -- 172.21.15.26:0/303502814 wait complete. 2024-03-24T10:41:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:inline data enabled 2024-03-24T10:41:18.403 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:18 smithi031 ceph-mon[37345]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:18.404 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:18 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/303502814' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:41:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:18 smithi026 ceph-mon[29210]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:18.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:18 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/303502814' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:41:18.848 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:41:18.858 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:18.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T10:41:19.080 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/303502814' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2 up:standby 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:19 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/303502814' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 2 up:standby 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:19.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:19 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:20.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:20 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.31:6826/1024705930,v1:172.21.15.31:6827/1024705930] up:boot 2024-03-24T10:41:20.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:20 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 3 up:standby 2024-03-24T10:41:20.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:20 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi031.vjdtgj"}]: dispatch 2024-03-24T10:41:20.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:20 smithi026 ceph-mon[29210]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:20.412 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:20 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.31:6826/1024705930,v1:172.21.15.31:6827/1024705930] up:boot 2024-03-24T10:41:20.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:20 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 3 up:standby 2024-03-24T10:41:20.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:20 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi031.vjdtgj"}]: dispatch 2024-03-24T10:41:20.413 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:20 smithi031 ceph-mon[37345]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:21.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 -- 172.21.15.26:0/584745169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c101600 msgr2=0x7f910c1039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 --2- 172.21.15.26:0/584745169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c101600 0x7f910c1039f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f910400b0a0 tx=0x7f910400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 -- 172.21.15.26:0/584745169 shutdown_connections 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 --2- 172.21.15.26:0/584745169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c103f30 0x7f910c106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 --2- 172.21.15.26:0/584745169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c101600 0x7f910c1039f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 -- 172.21.15.26:0/584745169 >> 172.21.15.26:0/584745169 conn(0x7f910c0faf70 msgr2=0x7f910c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 -- 172.21.15.26:0/584745169 shutdown_connections 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.862+0000 7f9111620700 1 -- 172.21.15.26:0/584745169 wait complete. 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 Processor -- start 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 -- start start 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c101600 0x7f910c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c197000 con 0x7f910c103f30 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f9111620700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c197140 con 0x7f910c101600 2024-03-24T10:41:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54018/0 (socket says 172.21.15.26:54018) 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 -- 172.21.15.26:0/575914182 learned_addr learned my addr 172.21.15.26:0/575914182 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910affd700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c101600 0x7f910c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 -- 172.21.15.26:0/575914182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c101600 msgr2=0x7f910c1964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c101600 0x7f910c1964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 -- 172.21.15.26:0/575914182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9104009d00 con 0x7f910c103f30 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.863+0000 7f910a7fc700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90f400d900 tx=0x7f90f400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.864+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90f40041f0 con 0x7f910c103f30 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.864+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90f4004cf0 con 0x7f910c103f30 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.864+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c19bba0 con 0x7f910c103f30 2024-03-24T10:41:21.197 INFO:teuthology.orchestra.run.smithi026.stdout:e12 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 12 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-03-24T10:41:04.185237+0000 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-03-24T10:41:20.162354+0000 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 60 2024-03-24T10:41:21.198 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 47 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 1 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:in 0 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=14486} 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data enabled 2024-03-24T10:41:21.199 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 1 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi026.ryhyqm{0:14486} state up:replay seq 2 join_fscid=1 addr [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi031.cgovmq{-1:24295} state up:standby seq 1 addr [v2:172.21.15.31:6824/3250075277,v1:172.21.15.31:6825/3250075277] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.cephfs.smithi031.vjdtgj{-1:24307} state up:standby seq 1 addr [v2:172.21.15.31:6826/1024705930,v1:172.21.15.31:6827/1024705930] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.864+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90f400b750 con 0x7f910c103f30 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.864+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c19c0f0 con 0x7f910c103f30 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.865+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c04ea80 con 0x7f910c103f30 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.865+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f90f4004750 con 0x7f910c103f30 2024-03-24T10:41:21.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.866+0000 7f9103fff700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 0x7f90f806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.866+0000 7f910affd700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 0x7f90f806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.866+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f90f408c220 con 0x7f910c103f30 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.866+0000 7f910affd700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 0x7f90f806f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9104007820 tx=0x7f9104006000 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:20.870+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90f405a020 con 0x7f910c103f30 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.133+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f910c19c9c0 con 0x7f910c103f30 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.133+0000 7f9103fff700 1 -- 172.21.15.26:0/575914182 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 12 v12) v1 ==== 76+0+1643 (secure 0 0 0) 0x7f90f4054e60 con 0x7f910c103f30 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 msgr2=0x7f90f806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 0x7f90f806f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9104007820 tx=0x7f9104006000 comp rx=0 tx=0).stop 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 msgr2=0x7f910c1969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90f400d900 tx=0x7f90f400dc10 comp rx=0 tx=0).stop 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 shutdown_connections 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f90f806cce0 0x7f90f806f1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f910c101600 0x7f910c1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 --2- 172.21.15.26:0/575914182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f910c103f30 0x7f910c1969e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:21.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 >> 172.21.15.26:0/575914182 conn(0x7f910c0faf70 msgr2=0x7f910c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:21.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 shutdown_connections 2024-03-24T10:41:21.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:21.136+0000 7f9111620700 1 -- 172.21.15.26:0/575914182 wait complete. 2024-03-24T10:41:21.202 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 12 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:standby 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: Dropping low affinity active daemon mds.cephfs.smithi026.dfxoad in favor of higher affinity standby. 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: Replacing daemon mds.cephfs.smithi026.dfxoad as rank 0 with standby daemon mds.cephfs.smithi026.ryhyqm 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 3 up:standby 2024-03-24T10:41:21.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: osdmap e47: 6 total, 6 up, 6 in 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:replay} 2 up:standby 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:21 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/575914182' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:standby 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: Dropping low affinity active daemon mds.cephfs.smithi026.dfxoad in favor of higher affinity standby. 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: Replacing daemon mds.cephfs.smithi026.dfxoad as rank 0 with standby daemon mds.cephfs.smithi026.ryhyqm 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T10:41:21.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.dfxoad=up:active} 3 up:standby 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: osdmap e47: 6 total, 6 up, 6 in 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:replay} 2 up:standby 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:21.452 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:21 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/575914182' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:41:21.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-24T10:41:22.443 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:reconnect 2024-03-24T10:41:22.444 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6826/3412130735,v1:172.21.15.26:6827/3412130735] up:boot 2024-03-24T10:41:22.444 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.31:6824/3250075277,v1:172.21.15.31:6825/3250075277] up:standby 2024-03-24T10:41:22.444 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:reconnect} 3 up:standby 2024-03-24T10:41:22.444 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:22.444 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:22 smithi026 ceph-mon[29210]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 2 op/s 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:reconnect 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6826/3412130735,v1:172.21.15.26:6827/3412130735] up:boot 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.31:6824/3250075277,v1:172.21.15.31:6825/3250075277] up:standby 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:reconnect} 3 up:standby 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi026.dfxoad"}]: dispatch 2024-03-24T10:41:22.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:22 smithi031 ceph-mon[37345]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 2 op/s 2024-03-24T10:41:23.449 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:23 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:rejoin 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:23 smithi031 ceph-mon[37345]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:rejoin} 3 up:standby 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:23 smithi031 ceph-mon[37345]: daemon mds.cephfs.smithi026.ryhyqm is now active in filesystem cephfs as rank 0 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:23 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:rejoin 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:23 smithi026 ceph-mon[29210]: fsmap cephfs:1/1 {0=cephfs.smithi026.ryhyqm=up:rejoin} 3 up:standby 2024-03-24T10:41:23.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:23 smithi026 ceph-mon[29210]: daemon mds.cephfs.smithi026.ryhyqm is now active in filesystem cephfs as rank 0 2024-03-24T10:41:24.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T10:41:24.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:active 2024-03-24T10:41:24.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: mds.? [v2:172.21.15.31:6826/1024705930,v1:172.21.15.31:6827/1024705930] up:standby 2024-03-24T10:41:24.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: fsmap cephfs:1 {0=cephfs.smithi026.ryhyqm=up:active} 3 up:standby 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:24 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.26:6828/1687655212,v1:172.21.15.26:6829/1687655212] up:active 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: mds.? [v2:172.21.15.31:6826/1024705930,v1:172.21.15.31:6827/1024705930] up:standby 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: fsmap cephfs:1 {0=cephfs.smithi026.ryhyqm=up:active} 3 up:standby 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T10:41:24.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:24 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:26.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:26 smithi031 ceph-mon[37345]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T10:41:26.338 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:26 smithi026 ceph-mon[29210]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T10:41:26.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 -- 172.21.15.26:0/28675702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568101ce0 msgr2=0x7f1568102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:26.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 --2- 172.21.15.26:0/28675702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568101ce0 0x7f1568102140 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f155c009910 tx=0x7f155c009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 -- 172.21.15.26:0/28675702 shutdown_connections 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 --2- 172.21.15.26:0/28675702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568101ce0 0x7f1568102140 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 --2- 172.21.15.26:0/28675702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1568100ae0 0x7f1568100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 -- 172.21.15.26:0/28675702 >> 172.21.15.26:0/28675702 conn(0x7f15680fc0a0 msgr2=0x7f15680fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 -- 172.21.15.26:0/28675702 shutdown_connections 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.357+0000 7f15706f6700 1 -- 172.21.15.26:0/28675702 wait complete. 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 Processor -- start 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 -- start start 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1568101ce0 0x7f15680731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15680736e0 con 0x7f1568100ae0 2024-03-24T10:41:26.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f15706f6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1568073820 con 0x7f1568101ce0 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37864/0 (socket says 172.21.15.26:37864) 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 -- 172.21.15.26:0/485351859 learned_addr learned my addr 172.21.15.26:0/485351859 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 -- 172.21.15.26:0/485351859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1568101ce0 msgr2=0x7f15680731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1568101ce0 0x7f15680731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 -- 172.21.15.26:0/485351859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0095f0 con 0x7f1568100ae0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.358+0000 7f156e492700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1558007e90 tx=0x7f155800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.359+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1558017070 con 0x7f1568100ae0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.359+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1568073b00 con 0x7f1568100ae0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.359+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f155800f040 con 0x7f1568100ae0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.359+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15580048f0 con 0x7f1568100ae0 2024-03-24T10:41:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.359+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156819e520 con 0x7f1568100ae0 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.360+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1568190660 con 0x7f1568100ae0 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.360+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f1558004120 con 0x7f1568100ae0 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.360+0000 7f15677fe700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 0x7f155006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.361+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f155808bd10 con 0x7f1568100ae0 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.361+0000 7f156dc91700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 0x7f155006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.361+0000 7f156dc91700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 0x7f155006f140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f155c00b5c0 tx=0x7f155c005160 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:26.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.365+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1558059b10 con 0x7f1568100ae0 2024-03-24T10:41:26.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.620+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f156804ea80 con 0x7f1568100ae0 2024-03-24T10:41:26.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.621+0000 7f15677fe700 1 -- 172.21.15.26:0/485351859 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4825 (secure 0 0 0) 0x7f156804ea80 con 0x7f1568100ae0 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 msgr2=0x7f155006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 0x7f155006f140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f155c00b5c0 tx=0x7f155c005160 comp rx=0 tx=0).stop 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 msgr2=0x7f1568074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1558007e90 tx=0x7f155800d3b0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 shutdown_connections 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f155006cc80 0x7f155006f140 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1568100ae0 0x7f1568074b00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 --2- 172.21.15.26:0/485351859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1568101ce0 0x7f15680731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 >> 172.21.15.26:0/485351859 conn(0x7f15680fc0a0 msgr2=0x7f1568105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 shutdown_connections 2024-03-24T10:41:26.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:26.623+0000 7f15706f6700 1 -- 172.21.15.26:0/485351859 wait complete. 2024-03-24T10:41:26.625 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 15 2024-03-24T10:41:26.634 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-03-24T10:41:27.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-24T10:41:27.556 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.687 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/485351859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:27.688 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/485351859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:27.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:28.927 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:28 smithi026 ceph-mon[29210]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T10:41:28.949 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:28 smithi031 ceph-mon[37345]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T10:41:29.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 -- 172.21.15.26:0/1578459908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 msgr2=0x7f2fa40ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:29.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 --2- 172.21.15.26:0/1578459908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa40ff000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009910 tx=0x7f2f94009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 -- 172.21.15.26:0/1578459908 shutdown_connections 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 --2- 172.21.15.26:0/1578459908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa40ff000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 --2- 172.21.15.26:0/1578459908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 0x7f2fa4102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 -- 172.21.15.26:0/1578459908 >> 172.21.15.26:0/1578459908 conn(0x7f2fa40fa720 msgr2=0x7f2fa40fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 -- 172.21.15.26:0/1578459908 shutdown_connections 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.701+0000 7f2fab941700 1 -- 172.21.15.26:0/1578459908 wait complete. 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 Processor -- start 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 -- start start 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 0x7f2fa40ffe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa41003b0 con 0x7f2fa40feba0 2024-03-24T10:41:29.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fab941700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa41004f0 con 0x7f2fa4102120 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37876/0 (socket says 172.21.15.26:37876) 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 -- 172.21.15.26:0/1766628661 learned_addr learned my addr 172.21.15.26:0/1766628661 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa8edc700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 0x7f2fa40ffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 -- 172.21.15.26:0/1766628661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 msgr2=0x7f2fa40ffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 0x7f2fa40ffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.702+0000 7f2fa96dd700 1 -- 172.21.15.26:0/1766628661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940095f0 con 0x7f2fa40feba0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2fa96dd700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000b9c0 tx=0x7f2fa000bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fa000d8b0 con 0x7f2fa40feba0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa41007d0 con 0x7f2fa40feba0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fa0015420 con 0x7f2fa40feba0 2024-03-24T10:41:29.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fa000c610 con 0x7f2fa40feba0 2024-03-24T10:41:29.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.703+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa419e650 con 0x7f2fa40feba0 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.704+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4190720 con 0x7f2fa40feba0 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.705+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f2fa0015590 con 0x7f2fa40feba0 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.705+0000 7f2f9a7fc700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 0x7f2f9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.705+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2fa0011070 con 0x7f2fa40feba0 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.705+0000 7f2fa8edc700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 0x7f2f9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:29.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.706+0000 7f2fa8edc700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 0x7f2f9006f1a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005090 tx=0x7f2f94005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:29.712+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2fa004f9f0 con 0x7f2fa40feba0 2024-03-24T10:41:30.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.002+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f2fa404ea80 con 0x7f2fa40feba0 2024-03-24T10:41:30.003 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.003 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:30.003 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.003 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.002+0000 7f2f9a7fc700 1 -- 172.21.15.26:0/1766628661 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+97 (secure 0 0 0) 0x7f2fa0059b40 con 0x7f2fa40feba0 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 msgr2=0x7f2f9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 0x7f2f9006f1a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005090 tx=0x7f2f94005960 comp rx=0 tx=0).stop 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 msgr2=0x7f2fa41017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000b9c0 tx=0x7f2fa000bcd0 comp rx=0 tx=0).stop 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 shutdown_connections 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2f9006cce0 0x7f2f9006f1a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2fa40feba0 0x7f2fa41017d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 --2- 172.21.15.26:0/1766628661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fa4102120 0x7f2fa40ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.004+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 >> 172.21.15.26:0/1766628661 conn(0x7f2fa40fa720 msgr2=0x7f2fa4105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:30.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.005+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 shutdown_connections 2024-03-24T10:41:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:30.005+0000 7f2fab941700 1 -- 172.21.15.26:0/1766628661 wait complete. 2024-03-24T10:41:30.016 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-03-24T10:41:30.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:30.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:30.887 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:30 smithi026 ceph-mon[29210]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T10:41:30.887 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:30 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1766628661' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T10:41:30.985 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:30 smithi031 ceph-mon[37345]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T10:41:30.986 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:30 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1766628661' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T10:41:31.154 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-24T10:41:31.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 -- 172.21.15.26:0/3363596626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d80fee60 msgr2=0x7fa3d80ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/3363596626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d80fee60 0x7fa3d80ff280 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009880 tx=0x7fa3c802bce0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 -- 172.21.15.26:0/3363596626 shutdown_connections 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/3363596626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d8100040 0x7fa3d81004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/3363596626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d80fee60 0x7fa3d80ff280 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 -- 172.21.15.26:0/3363596626 >> 172.21.15.26:0/3363596626 conn(0x7fa3d80fa440 msgr2=0x7fa3d80fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 -- 172.21.15.26:0/3363596626 shutdown_connections 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.298+0000 7fa3dd72e640 1 -- 172.21.15.26:0/3363596626 wait complete. 2024-03-24T10:41:31.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 Processor -- start 2024-03-24T10:41:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 -- start start 2024-03-24T10:41:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d80fee60 0x7fa3d8195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8196630 con 0x7fa3d8100040 2024-03-24T10:41:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3dd72e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81967a0 con 0x7fa3d80fee60 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37900/0 (socket says 172.21.15.26:37900) 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 -- 172.21.15.26:0/2696365042 learned_addr learned my addr 172.21.15.26:0/2696365042 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d6ffd640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d80fee60 0x7fa3d8195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 -- 172.21.15.26:0/2696365042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d80fee60 msgr2=0x7fa3d8195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d80fee60 0x7fa3d8195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 -- 172.21.15.26:0/2696365042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8009560 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.299+0000 7fa3d67fc640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00d6f0 tx=0x7fa3cc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.300+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3cc009a30 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.300+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3cc0054e0 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.300+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d819b1e0 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.300+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3cc010460 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.300+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d819b7c0 con 0x7fa3d8100040 2024-03-24T10:41:31.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.301+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d8107a10 con 0x7fa3d8100040 2024-03-24T10:41:31.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.301+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa3cc009b90 con 0x7fa3d8100040 2024-03-24T10:41:31.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.301+0000 7fa3bffff640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 0x7fa3c006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.302+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa3cc08be50 con 0x7fa3d8100040 2024-03-24T10:41:31.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.302+0000 7fa3d6ffd640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 0x7fa3c006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.302+0000 7fa3d6ffd640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 0x7fa3c006f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa3c802c7c0 tx=0x7fa3c8005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.306+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa3cc0561c0 con 0x7fa3d8100040 2024-03-24T10:41:31.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.492+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3d8107bd0 con 0x7fa3d8100040 2024-03-24T10:41:31.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.493+0000 7fa3bffff640 1 -- 172.21.15.26:0/2696365042 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15090 (secure 0 0 0) 0x7fa3cc0597e0 con 0x7fa3d8100040 2024-03-24T10:41:31.493 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":47,"fsid":"e07c400a-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:33:54.305014+0000","modified":"2024-03-24T10:41:20.162024+0000","last_up_change":"2024-03-24T10:39:45.361543+0000","last_in_change":"2024-03-24T10:39:26.209955+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:38:31.976899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-24T10:41:02.563345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-24T10:41:03.424548+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"20a15cb6-4489-4794-a110-50a62c92cc96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6803","nonce":4085968790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6805","nonce":4085968790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6809","nonce":4085968790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":4085968790},{"type":"v1","addr":"172.21.15.26:6807","nonce":4085968790}]},"public_addr":"172.21.15.26:6803/4085968790","cluster_addr":"172.21.15.26:6805/4085968790","heartbeat_back_addr":"172.21.15.26:6809/4085968790","heartbeat_front_addr":"172.21.15.26:6807/4085968790","state":["exists","up"]},{"osd":1,"uuid":"d1ad8eb2-18e8-4012-93f5-417d61a2832e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6811","nonce":551068105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6813","nonce":551068105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6817","nonce":551068105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":551068105},{"type":"v1","addr":"172.21.15.26:6815","nonce":551068105}]},"public_addr":"172.21.15.26:6811/551068105","cluster_addr":"172.21.15.26:6813/551068105","heartbeat_back_addr":"172.21.15.26:6817/551068105","heartbeat_front_addr":"172.21.15.26:6815/551068105","state":["exists","up"]},{"osd":2,"uuid":"c2ac73fc-bfe2-4b67-a75b-c5c77df7f8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6819","nonce":291489572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6821","nonce":291489572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6825","nonce":291489572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":291489572},{"type":"v1","addr":"172.21.15.26:6823","nonce":291489572}]},"public_addr":"172.21.15.26:6819/291489572","cluster_addr":"172.21.15.26:6821/291489572","heartbeat_back_addr":"172.21.15.26:6825/291489572","heartbeat_front_addr":"172.21.15.26:6823/291489572","state":["exists","up"]},{"osd":3,"uuid":"fd6ae4bc-dd6b-4dbc-ac84-b10495658433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6801","nonce":2495297239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6803","nonce":2495297239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6807","nonce":2495297239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2495297239},{"type":"v1","addr":"172.21.15.31:6805","nonce":2495297239}]},"public_addr":"172.21.15.31:6801/2495297239","cluster_addr":"172.21.15.31:6803/2495297239","heartbeat_back_addr":"172.21.15.31:6807/2495297239","heartbeat_front_addr":"172.21.15.31:6805/2495297239","state":["exists","up"]},{"osd":4,"uuid":"0e08bc7e-2981-441e-957a-6810a298651b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6809","nonce":3345716330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6811","nonce":3345716330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6815","nonce":3345716330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3345716330},{"type":"v1","addr":"172.21.15.31:6813","nonce":3345716330}]},"public_addr":"172.21.15.31:6809/3345716330","cluster_addr":"172.21.15.31:6811/3345716330","heartbeat_back_addr":"172.21.15.31:6815/3345716330","heartbeat_front_addr":"172.21.15.31:6813/3345716330","state":["exists","up"]},{"osd":5,"uuid":"956ded5d-73fa-44a5-9442-3f02c11d0683","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6817","nonce":2153668083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6819","nonce":2153668083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6823","nonce":2153668083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2153668083},{"type":"v1","addr":"172.21.15.31:6821","nonce":2153668083}]},"public_addr":"172.21.15.31:6817/2153668083","cluster_addr":"172.21.15.31:6819/2153668083","heartbeat_back_addr":"172.21.15.31:6823/2153668083","heartbeat_front_addr":"172.21.15.31:6821/2153668083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:32.191494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:01.114679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:28.646628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:38:52.463191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:18.061331+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:39:43.503559+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6827/2645357503":"2024-03-25T10:41:20.161984+0000","172.21.15.26:0/787137094":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3817743035":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6800/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/393242092":"2024-03-25T10:36:13.638224+0000","172.21.15.26:0/2873124826":"2024-03-25T10:34:30.200722+0000","172.21.15.26:0/105140095":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6801/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3967355473":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/884630952":"2024-03-25T10:35:03.260951+0000","172.21.15.26:0/3322398799":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6801/4217413357":"2024-03-25T10:34:30.200722+0000","172.21.15.26:6800/3234553684":"2024-03-25T10:35:03.260951+0000","172.21.15.26:6826/2645357503":"2024-03-25T10:41:20.161984+0000","172.21.15.26:6800/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:6801/1986399491":"2024-03-25T10:36:13.638224+0000","172.21.15.26:0/2057585064":"2024-03-25T10:36:13.638224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:41:31.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 msgr2=0x7fa3c006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 0x7fa3c006f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa3c802c7c0 tx=0x7fa3c8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 msgr2=0x7fa3d8196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00d6f0 tx=0x7fa3cc00dbb0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 shutdown_connections 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa3c006cc50 0x7fa3c006f110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d8100040 0x7fa3d8196060 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 --2- 172.21.15.26:0/2696365042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3d80fee60 0x7fa3d8195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 >> 172.21.15.26:0/2696365042 conn(0x7fa3d80fa440 msgr2=0x7fa3d80672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 shutdown_connections 2024-03-24T10:41:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.495+0000 7fa3dd72e640 1 -- 172.21.15.26:0/2696365042 wait complete. 2024-03-24T10:41:31.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:38:31.976899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-24T10:41:02.563345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-24T10:41:03.424548+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:41:31.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:41:31.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 -- 172.21.15.26:0/3769509947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 msgr2=0x7f9ed40ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3769509947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed40ff2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009880 tx=0x7f9ecc02bce0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 -- 172.21.15.26:0/3769509947 shutdown_connections 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3769509947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3769509947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed40ff2c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 -- 172.21.15.26:0/3769509947 >> 172.21.15.26:0/3769509947 conn(0x7f9ed40fa450 msgr2=0x7f9ed40fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 -- 172.21.15.26:0/3769509947 shutdown_connections 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.613+0000 7f9ed9977640 1 -- 172.21.15.26:0/3769509947 wait complete. 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 Processor -- start 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 -- start start 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4103350 con 0x7f9ed40feea0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed9977640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed41019d0 con 0x7f9ed40ffff0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37924/0 (socket says 172.21.15.26:37924) 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 -- 172.21.15.26:0/3192633216 learned_addr learned my addr 172.21.15.26:0/3192633216 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed27fc640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 -- 172.21.15.26:0/3192633216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 msgr2=0x7f9ed4101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed2ffd640 1 -- 172.21.15.26:0/3192633216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc009560 con 0x7f9ed40feea0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.614+0000 7f9ed27fc640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed2ffd640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc036040 tx=0x7f9ecc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ecc004520 con 0x7f9ed40feea0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ecc003ab0 con 0x7f9ed40feea0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed4101c50 con 0x7f9ed40feea0 2024-03-24T10:41:31.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ecc005670 con 0x7f9ed40feea0 2024-03-24T10:41:31.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.615+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed41021d0 con 0x7f9ed40feea0 2024-03-24T10:41:31.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.616+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed4107a10 con 0x7f9ed40feea0 2024-03-24T10:41:31.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.616+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f9ecc003c20 con 0x7f9ed40feea0 2024-03-24T10:41:31.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.617+0000 7f9ed8975640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 0x7f9ec006f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.617+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9ecc0ae660 con 0x7f9ed40feea0 2024-03-24T10:41:31.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.617+0000 7f9ed27fc640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 0x7f9ec006f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.617+0000 7f9ed27fc640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 0x7f9ec006f0b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4102af0 tx=0x7f9ebc005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.621+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9ecc0789d0 con 0x7f9ed40feea0 2024-03-24T10:41:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.802+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ed419dd90 con 0x7f9ed40feea0 2024-03-24T10:41:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.802+0000 7f9ed8975640 1 -- 172.21.15.26:0/3192633216 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ecc07bff0 con 0x7f9ed40feea0 2024-03-24T10:41:31.803 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-03-24T10:41:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 msgr2=0x7f9ec006f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 0x7f9ec006f0b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4102af0 tx=0x7f9ebc005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 msgr2=0x7f9ed4102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc036040 tx=0x7f9ecc005dc0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 shutdown_connections 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f9ec006cbf0 0x7f9ec006f0b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed40ffff0 0x7f9ed4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 --2- 172.21.15.26:0/3192633216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ed40feea0 0x7f9ed4102d80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 >> 172.21.15.26:0/3192633216 conn(0x7f9ed40fa450 msgr2=0x7f9ed4068430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 shutdown_connections 2024-03-24T10:41:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.804+0000 7f9ed9977640 1 -- 172.21.15.26:0/3192633216 wait complete. 2024-03-24T10:41:31.815 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:31 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2696365042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:41:31.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.930+0000 7ff086116640 1 -- 172.21.15.26:0/3442455134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0801011b0 msgr2=0x7ff080101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.930+0000 7ff086116640 1 --2- 172.21.15.26:0/3442455134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0801011b0 0x7ff080101630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff068009880 tx=0x7ff06802bce0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- 172.21.15.26:0/3442455134 shutdown_connections 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 --2- 172.21.15.26:0/3442455134 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0801011b0 0x7ff080101630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 --2- 172.21.15.26:0/3442455134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0800fffc0 0x7ff0801003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- 172.21.15.26:0/3442455134 >> 172.21.15.26:0/3442455134 conn(0x7ff0800fb530 msgr2=0x7ff0800fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- 172.21.15.26:0/3442455134 shutdown_connections 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- 172.21.15.26:0/3442455134 wait complete. 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 Processor -- start 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- start start 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0801011b0 0x7ff080195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.931+0000 7ff086116640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080196520 con 0x7ff0800fffc0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff086116640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080196690 con 0x7ff0801011b0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37948/0 (socket says 172.21.15.26:37948) 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 -- 172.21.15.26:0/2864606718 learned_addr learned my addr 172.21.15.26:0/2864606718 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07effd640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0801011b0 0x7ff080195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 -- 172.21.15.26:0/2864606718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0801011b0 msgr2=0x7ff080195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0801011b0 0x7ff080195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 -- 172.21.15.26:0/2864606718 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068009560 con 0x7ff0800fffc0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07f7fe640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff07000cc30 tx=0x7ff07000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff070003b40 con 0x7ff0800fffc0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff07000ce70 con 0x7ff0800fffc0 2024-03-24T10:41:31.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08019b110 con 0x7ff0800fffc0 2024-03-24T10:41:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff07000f7e0 con 0x7ff0800fffc0 2024-03-24T10:41:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.932+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08019b6c0 con 0x7ff0800fffc0 2024-03-24T10:41:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.933+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08004e030 con 0x7ff0800fffc0 2024-03-24T10:41:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.934+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ff070003ca0 con 0x7ff0800fffc0 2024-03-24T10:41:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.934+0000 7ff07cff9640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 0x7ff06c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.934+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff07008c410 con 0x7ff0800fffc0 2024-03-24T10:41:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.934+0000 7ff07effd640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 0x7ff06c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.934+0000 7ff07effd640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 0x7ff06c06f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff068009230 tx=0x7ff068005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:31.939+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff070056830 con 0x7ff0800fffc0 2024-03-24T10:41:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.120+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff08018fb30 con 0x7ff0800fffc0 2024-03-24T10:41:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.120+0000 7ff07cff9640 1 -- 172.21.15.26:0/2864606718 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff070059e50 con 0x7ff0800fffc0 2024-03-24T10:41:32.121 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-03-24T10:41:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 msgr2=0x7ff06c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 0x7ff06c06f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff068009230 tx=0x7ff068005fd0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 msgr2=0x7ff080195aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff07000cc30 tx=0x7ff07000a620 comp rx=0 tx=0).stop 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 shutdown_connections 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff06c06cc50 0x7ff06c06f110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0801011b0 0x7ff080195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 --2- 172.21.15.26:0/2864606718 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0800fffc0 0x7ff080195aa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 >> 172.21.15.26:0/2864606718 conn(0x7ff0800fb530 msgr2=0x7ff080104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 shutdown_connections 2024-03-24T10:41:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.123+0000 7ff086116640 1 -- 172.21.15.26:0/2864606718 wait complete. 2024-03-24T10:41:32.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-24T10:41:32.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:31 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2696365042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:41:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 -- 172.21.15.26:0/3967973453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80feea0 msgr2=0x7efff80ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 --2- 172.21.15.26:0/3967973453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80feea0 0x7efff80ff2c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7effe8009880 tx=0x7effe802bce0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 -- 172.21.15.26:0/3967973453 shutdown_connections 2024-03-24T10:41:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 --2- 172.21.15.26:0/3967973453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80ffff0 0x7efff8100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 --2- 172.21.15.26:0/3967973453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80feea0 0x7efff80ff2c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 -- 172.21.15.26:0/3967973453 >> 172.21.15.26:0/3967973453 conn(0x7efff80fa430 msgr2=0x7efff80fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 -- 172.21.15.26:0/3967973453 shutdown_connections 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 -- 172.21.15.26:0/3967973453 wait complete. 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.244+0000 7efffdb69640 1 Processor -- start 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efffdb69640 1 -- start start 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efffdb69640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80feea0 0x7efff8067d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efffdb69640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efffdb69640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff819adf0 con 0x7efff80ffff0 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efffdb69640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff819af30 con 0x7efff80feea0 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37978/0 (socket says 172.21.15.26:37978) 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 -- 172.21.15.26:0/270940025 learned_addr learned my addr 172.21.15.26:0/270940025 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff77fe640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80feea0 0x7efff8067d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 -- 172.21.15.26:0/270940025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80feea0 msgr2=0x7efff8067d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80feea0 0x7efff8067d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 -- 172.21.15.26:0/270940025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8009560 con 0x7efff80ffff0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.245+0000 7efff6ffd640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7effe400d8d0 tx=0x7effe400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.246+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe4009a30 con 0x7efff80ffff0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.246+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff819b1c0 con 0x7efff80ffff0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.246+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effe4009b90 con 0x7efff80ffff0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.246+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe400f680 con 0x7efff80ffff0 2024-03-24T10:41:32.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.246+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff819b7a0 con 0x7efff80ffff0 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.247+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff818e9a0 con 0x7efff80ffff0 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.247+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7effe400f900 con 0x7efff80ffff0 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.248+0000 7efff4ff9640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 0x7effdc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.248+0000 7efff77fe640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 0x7effdc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.248+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7effe408c050 con 0x7efff80ffff0 2024-03-24T10:41:32.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.248+0000 7efff77fe640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 0x7effdc06f110 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7effe8009230 tx=0x7effe8005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.252+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7effe4056470 con 0x7efff80ffff0 2024-03-24T10:41:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.435+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7efff8107a10 con 0x7efff80ffff0 2024-03-24T10:41:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.435+0000 7efff4ff9640 1 -- 172.21.15.26:0/270940025 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7effe4059a90 con 0x7efff80ffff0 2024-03-24T10:41:32.436 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-03-24T10:41:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 msgr2=0x7effdc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 0x7effdc06f110 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7effe8009230 tx=0x7effe8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 msgr2=0x7efff819a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7effe400d8d0 tx=0x7effe400dd90 comp rx=0 tx=0).stop 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 shutdown_connections 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7effdc06cc50 0x7effdc06f110 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efff80ffff0 0x7efff819a820 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 --2- 172.21.15.26:0/270940025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efff80feea0 0x7efff8067d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 >> 172.21.15.26:0/270940025 conn(0x7efff80fa430 msgr2=0x7efff81036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 shutdown_connections 2024-03-24T10:41:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.437+0000 7efffdb69640 1 -- 172.21.15.26:0/270940025 wait complete. 2024-03-24T10:41:32.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:41:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.555+0000 7f141469d640 1 -- 172.21.15.26:0/2428347927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 msgr2=0x7f140c1016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.555+0000 7f141469d640 1 --2- 172.21.15.26:0/2428347927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c1016c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009880 tx=0x7f13fc02bce0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- 172.21.15.26:0/2428347927 shutdown_connections 2024-03-24T10:41:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 --2- 172.21.15.26:0/2428347927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c1016c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 --2- 172.21.15.26:0/2428347927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 0x7f140c0fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- 172.21.15.26:0/2428347927 >> 172.21.15.26:0/2428347927 conn(0x7f140c0f9be0 msgr2=0x7f140c0fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- 172.21.15.26:0/2428347927 shutdown_connections 2024-03-24T10:41:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- 172.21.15.26:0/2428347927 wait complete. 2024-03-24T10:41:32.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 Processor -- start 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- start start 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 0x7f140c195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1965b0 con 0x7f140c101240 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.556+0000 7f141469d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c196720 con 0x7f140c0fe270 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38008/0 (socket says 172.21.15.26:38008) 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 -- 172.21.15.26:0/2633722929 learned_addr learned my addr 172.21.15.26:0/2633722929 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1412439640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 0x7f140c195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 -- 172.21.15.26:0/2633722929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 msgr2=0x7f140c195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 0x7f140c195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 -- 172.21.15.26:0/2633722929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408009490 con 0x7f140c101240 2024-03-24T10:41:32.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f1411c38640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13fc036040 tx=0x7f13fc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc004520 con 0x7f140c101240 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc009560 con 0x7f140c101240 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13fc003ab0 con 0x7f140c101240 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc005670 con 0x7f140c101240 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.557+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c19b4f0 con 0x7f140c101240 2024-03-24T10:41:32.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.558+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c18fb10 con 0x7f140c101240 2024-03-24T10:41:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.559+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f13fc003c20 con 0x7f140c101240 2024-03-24T10:41:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.559+0000 7f14037fe640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 0x7f13f806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.559+0000 7f1412439640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 0x7f13f806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.559+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f13fc0ae660 con 0x7f140c101240 2024-03-24T10:41:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.560+0000 7f1412439640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 0x7f13f806f0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f140c067040 tx=0x7f1408005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.564+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f13fc0789d0 con 0x7f140c101240 2024-03-24T10:41:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.794+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f140c108b80 con 0x7f140c101240 2024-03-24T10:41:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.795+0000 7f14037fe640 1 -- 172.21.15.26:0/2633722929 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4825 (secure 0 0 0) 0x7f13fc07bff0 con 0x7f140c101240 2024-03-24T10:41:32.795 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-24T10:41:32.795 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14498,"name":"cephfs.smithi026.dfxoad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6827/3412130735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":3412130735},{"type":"v1","addr":"172.21.15.26:6827","nonce":3412130735}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24295,"name":"cephfs.smithi031.cgovmq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.31:6825/3250075277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3250075277},{"type":"v1","addr":"172.21.15.31:6825","nonce":3250075277}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24307,"name":"cephfs.smithi031.vjdtgj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.31:6827/1024705930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":1024705930},{"type":"v1","addr":"172.21.15.31:6827","nonce":1024705930}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:41:04.185237+0000","modified":"2024-03-24T10:41:23.171950+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi026.ryhyqm","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.26:6829/1687655212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1687655212},{"type":"v1","addr":"172.21.15.26:6829","nonce":1687655212}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-24T10:41:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 15 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 msgr2=0x7f13f806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 0x7f13f806f0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f140c067040 tx=0x7f1408005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 msgr2=0x7f140c195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13fc036040 tx=0x7f13fc005dc0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 shutdown_connections 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f13f806cbf0 0x7f13f806f0b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f140c101240 0x7f140c195fe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 --2- 172.21.15.26:0/2633722929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c0fe270 0x7f140c195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 >> 172.21.15.26:0/2633722929 conn(0x7f140c0f9be0 msgr2=0x7f140c104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 shutdown_connections 2024-03-24T10:41:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:32.797+0000 7f141469d640 1 -- 172.21.15.26:0/2633722929 wait complete. 2024-03-24T10:41:32.808 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-03-24T10:41:32.809 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T10:41:32.834 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T10:41:32.834 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-24T10:41:32.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:32.834 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-03-24T10:41:32.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:32.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link delete ceph-brx 2024-03-24T10:41:32.920 INFO:teuthology.orchestra.run.smithi026.stderr:Cannot find device "ceph-brx" 2024-03-24T10:41:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:41:32.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:32.922 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-24T10:41:32.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:32.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-03-24T10:41:32.973 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:32 smithi026 ceph-mon[29210]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:32.973 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:32 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/3192633216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:41:32.973 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:32 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2864606718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T10:41:32.973 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:32 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/270940025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T10:41:33.010 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:32 smithi031 ceph-mon[37345]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:33.010 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:32 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/3192633216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:41:33.011 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:32 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2864606718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T10:41:33.011 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:32 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/270940025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T10:41:33.018 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-03-24T10:41:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:41:33.019 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-24T10:41:33.019 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:41:33.019 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:33.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T10:41:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:33.850 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:33 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2633722929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:34.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:33 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2633722929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:34.819 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:34.819 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:34.820 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:34.820 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:34.820 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:34.820 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:34 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 -- 172.21.15.26:0/2285780182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 msgr2=0x7ff48c100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 --2- 172.21.15.26:0/2285780182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c100f10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff474009910 tx=0x7ff474009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 -- 172.21.15.26:0/2285780182 shutdown_connections 2024-03-24T10:41:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 --2- 172.21.15.26:0/2285780182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c100f10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 --2- 172.21.15.26:0/2285780182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 -- 172.21.15.26:0/2285780182 >> 172.21.15.26:0/2285780182 conn(0x7ff48c0fc5a0 msgr2=0x7ff48c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 -- 172.21.15.26:0/2285780182 shutdown_connections 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.060+0000 7ff49080a700 1 -- 172.21.15.26:0/2285780182 wait complete. 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 Processor -- start 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 -- start start 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c19d2a0 con 0x7ff48c100a90 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff49080a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c196810 con 0x7ff48c106a90 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59586/0 (socket says 172.21.15.26:59586) 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 -- 172.21.15.26:0/2954767947 learned_addr learned my addr 172.21.15.26:0/2954767947 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff489d9b700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 -- 172.21.15.26:0/2954767947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 msgr2=0x7ff48c19cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c19cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff48a59c700 1 -- 172.21.15.26:0/2954767947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4740095f0 con 0x7ff48c100a90 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.061+0000 7ff489d9b700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff48a59c700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4840060b0 tx=0x7ff48400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff48400dda0 con 0x7ff48c100a90 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff48400f040 con 0x7ff48c100a90 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c196af0 con 0x7ff48c100a90 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff484014790 con 0x7ff48c100a90 2024-03-24T10:41:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.062+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c197040 con 0x7ff48c100a90 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.063+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c1090b0 con 0x7ff48c100a90 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.063+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ff484014980 con 0x7ff48c100a90 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.064+0000 7ff4837fe700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 0x7ff47806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.064+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff48408bd70 con 0x7ff48c100a90 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.064+0000 7ff489d9b700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 0x7ff47806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.064+0000 7ff489d9b700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 0x7ff47806f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff4740059d0 tx=0x7ff47401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.068+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff484059c20 con 0x7ff48c100a90 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:35.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:34 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:41:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.316+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff48c197d20 con 0x7ff48c100a90 2024-03-24T10:41:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.317+0000 7ff4837fe700 1 -- 172.21.15.26:0/2954767947 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7ff4840597b0 con 0x7ff48c100a90 2024-03-24T10:41:35.317 INFO:teuthology.orchestra.run.smithi026.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T10:41:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 msgr2=0x7ff47806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 0x7ff47806f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff4740059d0 tx=0x7ff47401a040 comp rx=0 tx=0).stop 2024-03-24T10:41:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 msgr2=0x7ff48c19c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4840060b0 tx=0x7ff48400d3b0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 shutdown_connections 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ff47806cce0 0x7ff47806f1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff48c100a90 0x7ff48c19c6d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 --2- 172.21.15.26:0/2954767947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff48c106a90 0x7ff48c19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 >> 172.21.15.26:0/2954767947 conn(0x7ff48c0fc5a0 msgr2=0x7ff48c0fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 shutdown_connections 2024-03-24T10:41:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:35.319+0000 7ff49080a700 1 -- 172.21.15.26:0/2954767947 wait complete. 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi026.front.sepia.ceph.com 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:41:35.927 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:41:35.928 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:41:35.928 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:41:35.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:35.928 DEBUG:teuthology.orchestra.run.smithi026:> ip addr 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 ::1/128 scope host 2024-03-24T10:41:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f0 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f1 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c0 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f0 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: inet 172.21.15.26/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft 42241sec preferred_lft 42241sec 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c1 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f1 2024-03-24T10:41:35.946 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add name ceph-brx type bridge 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr flush dev ceph-brx 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set ceph-brx up 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T10:41:35.946 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-03-24T10:41:36.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:36.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:36.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:36.152 DEBUG:teuthology.orchestra.run.smithi026:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:41:36.180 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-24T10:41:36.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:36.182 DEBUG:teuthology.orchestra.run.smithi026:> route 2024-03-24T10:41:36.254 INFO:teuthology.orchestra.run.smithi026.stdout:Kernel IP routing table 2024-03-24T10:41:36.254 INFO:teuthology.orchestra.run.smithi026.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:41:36.254 INFO:teuthology.orchestra.run.smithi026.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:41:36.254 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:41:36.255 INFO:teuthology.orchestra.run.smithi026.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:41:36.255 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-03-24T10:41:36.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:36.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:36 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/2954767947' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:41:36.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:36 smithi031 ceph-mon[37345]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:36 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/2954767947' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:41:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:36 smithi026 ceph-mon[29210]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T10:41:36.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:36.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:36.586 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-03-24T10:41:36.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:36.601 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list-id 2024-03-24T10:41:36.655 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:36.656 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-03-24T10:41:36.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:41:36.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T10:41:36.656 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-03-24T10:41:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:36.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:36.778 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T10:41:36.779 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-03-24T10:41:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:36.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:36.914 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:36.914 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-03-24T10:41:36.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set brx.0 up 2024-03-24T10:41:36.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T10:41:36.914 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-03-24T10:41:36.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:36.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:41:36.995 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:37.009 INFO:teuthology.orchestra.run.smithi026.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:41:37.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:41:37.010 DEBUG:teuthology.orchestra.run.smithi026:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:37.064 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T10:41:37.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:37.065 DEBUG:teuthology.orchestra.run.smithi026:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T10:41:37.160 INFO:teuthology.orchestra.run.smithi026.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T10:41:37.160 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T10:41:37.160 INFO:teuthology.orchestra.run.smithi026.stdout:invalid new device string format 2024-03-24T10:41:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T10:41:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T10:41:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:invalid new device string format 2024-03-24T10:41:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: resolved to: "172.21.15.26:3300,172.21.15.31:3300" 2024-03-24T10:41:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:invalid new device string format 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: resolved to: "172.21.15.26:3300,172.21.15.31:3300" 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: resolved to: "172.21.15.26:6789,172.21.15.31:6789" 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.26:6789,172.21.15.31:6789:/ 2024-03-24T10:41:37.285 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e07c400a-e9c9-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T10:41:37.286 INFO:teuthology.orchestra.run.smithi026.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T10:41:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: You just mounted a file system that supports labels which does not 2024-03-24T10:41:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T10:41:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T10:41:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T10:41:37.288 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T10:41:37.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:37.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:37.317 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:37.421 INFO:teuthology.orchestra.run.smithi026.stdout:client14536 2024-03-24T10:41:37.422 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:41:37.422 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:41:37.437 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:37.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:37.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:37.543 INFO:teuthology.orchestra.run.smithi026.stdout:client14536 2024-03-24T10:41:37.543 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:37.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:37.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:37.607 INFO:teuthology.orchestra.run.smithi026.stdout:client14536 2024-03-24T10:41:37.607 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715.client14536/status') 2024-03-24T10:41:37.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:37.675 INFO:teuthology.orchestra.run.smithi026.stdout:instance: client.14536 (3)192.168.0.1:0/2396857942 2024-03-24T10:41:37.675 INFO:teuthology.orchestra.run.smithi026.stdout:blocklisted: false 2024-03-24T10:41:37.675 INFO:teuthology.orchestra.run.smithi026.stderr:0+1 records in 2024-03-24T10:41:37.675 INFO:teuthology.orchestra.run.smithi026.stderr:0+1 records out 2024-03-24T10:41:37.675 INFO:teuthology.orchestra.run.smithi026.stderr:70 bytes copied, 2.3982e-05 s, 2.9 MB/s 2024-03-24T10:41:37.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:37.682 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-24T10:41:37.682 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:41:37.682 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:37.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T10:41:37.894 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:37.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:37 smithi026 ceph-mon[29210]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T10:41:38.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:37 smithi031 ceph-mon[37345]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T10:41:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- 172.21.15.26:0/1959400743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0739f0 msgr2=0x7f2e1c10cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 --2- 172.21.15.26:0/1959400743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c10cdf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009910 tx=0x7f2e0c009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- 172.21.15.26:0/1959400743 shutdown_connections 2024-03-24T10:41:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 --2- 172.21.15.26:0/1959400743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c10cdf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 --2- 172.21.15.26:0/1959400743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c0734b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- 172.21.15.26:0/1959400743 >> 172.21.15.26:0/1959400743 conn(0x7f2e1c0fc5a0 msgr2=0x7f2e1c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- 172.21.15.26:0/1959400743 shutdown_connections 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- 172.21.15.26:0/1959400743 wait complete. 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 Processor -- start 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.536+0000 7f2e23f89700 1 -- start start 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e23f89700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e23f89700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e23f89700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c19f4d0 con 0x7f2e1c0730d0 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e23f89700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c198a40 con 0x7f2e1c0739f0 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59614/0 (socket says 172.21.15.26:59614) 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 -- 172.21.15.26:0/78504782 learned_addr learned my addr 172.21.15.26:0/78504782 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21524700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 -- 172.21.15.26:0/78504782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 msgr2=0x7f2e1c19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 -- 172.21.15.26:0/78504782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0095f0 con 0x7f2e1c0730d0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21524700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e21d25700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800e920 tx=0x7f2e1800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e18009790 con 0x7f2e1c0730d0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e1800cba0 con 0x7f2e1c0730d0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c198d20 con 0x7f2e1c0730d0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.537+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e18018650 con 0x7f2e1c0730d0 2024-03-24T10:41:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.538+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c199270 con 0x7f2e1c0730d0 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.538+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c074810 con 0x7f2e1c0730d0 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.539+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f2e1800cd10 con 0x7f2e1c0730d0 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.539+0000 7f2e12ffd700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 0x7f2e0806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.539+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2e18014070 con 0x7f2e1c0730d0 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.539+0000 7f2e21524700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 0x7f2e0806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.540+0000 7f2e21524700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 0x7f2e0806f1b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b760 tx=0x7f2e0c009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:39.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.544+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e180529f0 con 0x7f2e1c0730d0 2024-03-24T10:41:39.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.791+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2e1c199f30 con 0x7f2e1c0730d0 2024-03-24T10:41:39.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.791+0000 7f2e12ffd700 1 -- 172.21.15.26:0/78504782 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f2e1805a190 con 0x7f2e1c0730d0 2024-03-24T10:41:39.792 INFO:teuthology.orchestra.run.smithi026.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 msgr2=0x7f2e0806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 0x7f2e0806f1b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b760 tx=0x7f2e0c009f70 comp rx=0 tx=0).stop 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 msgr2=0x7f2e1c19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800e920 tx=0x7f2e1800ece0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 shutdown_connections 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f2e0806ccf0 0x7f2e0806f1b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e1c0730d0 0x7f2e1c19e900 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 --2- 172.21.15.26:0/78504782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c0739f0 0x7f2e1c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.793+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 >> 172.21.15.26:0/78504782 conn(0x7f2e1c0fc5a0 msgr2=0x7f2e1c1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:39.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.794+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 shutdown_connections 2024-03-24T10:41:39.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:39.794+0000 7f2e23f89700 1 -- 172.21.15.26:0/78504782 wait complete. 2024-03-24T10:41:40.095 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:39 smithi026 ceph-mon[29210]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2024-03-24T10:41:40.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:39 smithi031 ceph-mon[37345]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:41:40.409 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T10:41:40.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:40.410 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-03-24T10:41:40.428 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42246sec preferred_lft 42246sec 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:41:40.429 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:41:40.430 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-03-24T10:41:40.430 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-03-24T10:41:40.430 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T10:41:40.430 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-24T10:41:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:40.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:40.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:40.638 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:41:40.665 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-03-24T10:41:40.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:40.667 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-03-24T10:41:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-03-24T10:41:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:41:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:41:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:41:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:41:40.739 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:40.740 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-24T10:41:40.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:41:40.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:41:40.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:41:40.740 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-24T10:41:40.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:40.837 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:40 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/78504782' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:41:41.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:41.037 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-24T10:41:41.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:41.052 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-03-24T10:41:41.108 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:41.108 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-24T10:41:41.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:41:41.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T10:41:41.108 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-24T10:41:41.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:41.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:40 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/78504782' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:41:41.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.230 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T10:41:41.230 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T10:41:41.231 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-24T10:41:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:41.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.367 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:41:41.367 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-24T10:41:41.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-03-24T10:41:41.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T10:41:41.367 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-24T10:41:41.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:41.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.448 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:41:41.448 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:41.462 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T10:41:41.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:41:41.462 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:41.516 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T10:41:41.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:41.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T10:41:41.614 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T10:41:41.614 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T10:41:41.615 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2024-03-24T10:41:41.742 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T10:41:41.742 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T10:41:41.742 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2024-03-24T10:41:41.742 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.26:3300,172.21.15.31:3300" 2024-03-24T10:41:41.742 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.26:3300,172.21.15.31:3300" 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.26:6789,172.21.15.31:6789" 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.26:6789,172.21.15.31:6789:/ 2024-03-24T10:41:41.761 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e07c400a-e9c9-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T10:41:41.762 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-24T10:41:41.762 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-03-24T10:41:41.762 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T10:41:41.762 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T10:41:41.762 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T10:41:41.764 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T10:41:41.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:41.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:41.791 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.887 INFO:teuthology.orchestra.run.smithi031.stdout:client14542 2024-03-24T10:41:41.887 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:41:41.887 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:41:41.903 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:41.949 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:41 smithi031 ceph-mon[37345]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-24T10:41:41.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:41.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:41.998 INFO:teuthology.orchestra.run.smithi031.stdout:client14542 2024-03-24T10:41:41.998 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:41:42.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:41:42.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:42.094 INFO:teuthology.orchestra.run.smithi031.stdout:client14542 2024-03-24T10:41:42.094 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715.client14542/status') 2024-03-24T10:41:42.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:41:42.154 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.14542 (3)172.21.15.31:0/1740615791 2024-03-24T10:41:42.154 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-03-24T10:41:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-03-24T10:41:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-03-24T10:41:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:71 bytes copied, 2.057e-05 s, 3.5 MB/s 2024-03-24T10:41:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-24T10:41:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:41:42.161 INFO:teuthology.run_tasks:Running task print... 2024-03-24T10:41:42.175 INFO:teuthology.task.print:**** done client 2024-03-24T10:41:42.175 INFO:teuthology.run_tasks:Running task parallel... 2024-03-24T10:41:42.184 INFO:teuthology.task.parallel:starting parallel... 2024-03-24T10:41:42.184 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T10:41:42.185 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T10:41:42.185 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T10:41:42.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T10:41:42.186 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T10:41:42.186 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-24T10:41:42.189 INFO:tasks.workunit:Pulling workunits from ref d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T10:41:42.189 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T10:41:42.189 INFO:tasks.workunit:timeout=3h 2024-03-24T10:41:42.189 INFO:tasks.workunit:cleanup=True 2024-03-24T10:41:42.190 DEBUG:teuthology.orchestra.run.smithi026:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:42.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:41 smithi026 ceph-mon[29210]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 9fh/159d Inode: 1 Links: 2 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-24 10:41:10.161882788 +0000 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-24 10:41:37.315388954 +0000 2024-03-24T10:41:42.210 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-24 10:41:10.161882788 +0000 2024-03-24T10:41:42.210 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T10:41:42.210 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T10:41:42.284 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-24 10:41:42.278318217 +0000 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-24 10:41:42.278318217 +0000 2024-03-24T10:41:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-24 10:41:10.161882788 +0000 2024-03-24T10:41:42.302 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-24T10:41:42.302 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-24T10:41:42.372 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T10:41:42.373 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T10:41:42.388 INFO:tasks.workunit.client.0.smithi026.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T10:41:42.415 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:42.427 INFO:tasks.workunit.client.1.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- 172.21.15.26:0/1011196973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448100ad0 msgr2=0x7fa448100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 --2- 172.21.15.26:0/1011196973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448100ad0 0x7fa448100ef0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009910 tx=0x7fa43c009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- 172.21.15.26:0/1011196973 shutdown_connections 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 --2- 172.21.15.26:0/1011196973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448101cd0 0x7fa448102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 --2- 172.21.15.26:0/1011196973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448100ad0 0x7fa448100ef0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- 172.21.15.26:0/1011196973 >> 172.21.15.26:0/1011196973 conn(0x7fa4480fc070 msgr2=0x7fa4480fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- 172.21.15.26:0/1011196973 shutdown_connections 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- 172.21.15.26:0/1011196973 wait complete. 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 Processor -- start 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- start start 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 0x7fa448074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480736e0 con 0x7fa448101cd0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.977+0000 7fa44eef1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa448073820 con 0x7fa448100ad0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46420/0 (socket says 172.21.15.26:46420) 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 -- 172.21.15.26:0/161268885 learned_addr learned my addr 172.21.15.26:0/161268885 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa44cc8d700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 0x7fa448074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 -- 172.21.15.26:0/161268885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 msgr2=0x7fa448074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 0x7fa448074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 -- 172.21.15.26:0/161268885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c0095f0 con 0x7fa448101cd0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa44cc8d700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 0x7fa448074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa447fff700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d8d0 tx=0x7fa43400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa434009940 con 0x7fa448101cd0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa434010460 con 0x7fa448101cd0 2024-03-24T10:41:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43400f5f0 con 0x7fa448101cd0 2024-03-24T10:41:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa448073b00 con 0x7fa448101cd0 2024-03-24T10:41:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.978+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44819e510 con 0x7fa448101cd0 2024-03-24T10:41:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.979+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa448190650 con 0x7fa448101cd0 2024-03-24T10:41:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.980+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa4340105d0 con 0x7fa448101cd0 2024-03-24T10:41:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.980+0000 7fa445ffb700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 0x7fa43806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.980+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa43408c2b0 con 0x7fa448101cd0 2024-03-24T10:41:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.980+0000 7fa44cc8d700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 0x7fa43806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.981+0000 7fa44cc8d700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 0x7fa43806f1b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa43c006010 tx=0x7fa43c00b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:43.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:43.985+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa43405a0b0 con 0x7fa448101cd0 2024-03-24T10:41:44.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.187+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa44802ccb0 con 0x7fa448101cd0 2024-03-24T10:41:44.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:43 smithi026 ceph-mon[29210]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-24T10:41:44.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:43 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:44.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.188+0000 7fa445ffb700 1 -- 172.21.15.26:0/161268885 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa434059c40 con 0x7fa448101cd0 2024-03-24T10:41:44.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 msgr2=0x7fa43806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 0x7fa43806f1b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa43c006010 tx=0x7fa43c00b560 comp rx=0 tx=0).stop 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 msgr2=0x7fa4480731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d8d0 tx=0x7fa43400dc90 comp rx=0 tx=0).stop 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 shutdown_connections 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fa43806ccf0 0x7fa43806f1b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100ad0 0x7fa448074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 --2- 172.21.15.26:0/161268885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa448101cd0 0x7fa4480731a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 >> 172.21.15.26:0/161268885 conn(0x7fa4480fc070 msgr2=0x7fa4481052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 shutdown_connections 2024-03-24T10:41:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:44.190+0000 7fa44eef1700 1 -- 172.21.15.26:0/161268885 wait complete. 2024-03-24T10:41:44.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:43 smithi031 ceph-mon[37345]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-24T10:41:44.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:43 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:44.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T10:41:45.063 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:46.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:46 smithi031 ceph-mon[37345]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-24T10:41:46.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:46 smithi026 ceph-mon[29210]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-24T10:41:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.745+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/361623635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a88103ed0 msgr2=0x7f8a881062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.745+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/361623635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a88103ed0 0x7f8a881062c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009910 tx=0x7f8a7c009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.745+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/361623635 shutdown_connections 2024-03-24T10:41:46.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.745+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/361623635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a88103ed0 0x7f8a881062c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.745+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/361623635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a881015a0 0x7f8a88103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.746+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/361623635 >> 172.21.15.26:0/361623635 conn(0x7f8a880faf70 msgr2=0x7f8a880fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.746+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/361623635 shutdown_connections 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.746+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/361623635 wait complete. 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.746+0000 7f8a8e2e5700 1 Processor -- start 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.746+0000 7f8a8e2e5700 1 -- start start 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a8e2e5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a8e2e5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a88103ed0 0x7f8a880ff550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a8e2e5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880ffa90 con 0x7f8a881015a0 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a8e2e5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880ffbd0 con 0x7f8a88103ed0 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46448/0 (socket says 172.21.15.26:46448) 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 -- 172.21.15.26:0/3037926818 learned_addr learned my addr 172.21.15.26:0/3037926818 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:46.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a877fe700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a88103ed0 0x7f8a880ff550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 -- 172.21.15.26:0/3037926818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a88103ed0 msgr2=0x7f8a880ff550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a88103ed0 0x7f8a880ff550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 -- 172.21.15.26:0/3037926818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c0095f0 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.747+0000 7f8a87fff700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800d900 tx=0x7f8a7800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.748+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a780041f0 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.748+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a78004cf0 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.748+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a880ffeb0 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.748+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a7800b750 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.748+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8819e4d0 con 0x7f8a881015a0 2024-03-24T10:41:46.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.749+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8804ea80 con 0x7f8a881015a0 2024-03-24T10:41:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.750+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f8a78004750 con 0x7f8a881015a0 2024-03-24T10:41:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.750+0000 7f8a857fa700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 0x7f8a7006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.750+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8a7808c140 con 0x7f8a881015a0 2024-03-24T10:41:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.750+0000 7f8a877fe700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 0x7f8a7006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.751+0000 7f8a877fe700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 0x7f8a7006f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005090 tx=0x7f8a7c005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:46.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.755+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8a78059fc0 con 0x7f8a881015a0 2024-03-24T10:41:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.982+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f8a8819e660 con 0x7f8a881015a0 2024-03-24T10:41:46.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.982+0000 7f8a857fa700 1 -- 172.21.15.26:0/3037926818 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f8a78059b50 con 0x7f8a881015a0 2024-03-24T10:41:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 msgr2=0x7f8a7006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 0x7f8a7006f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005090 tx=0x7f8a7c005960 comp rx=0 tx=0).stop 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 msgr2=0x7f8a88100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800d900 tx=0x7f8a7800dc10 comp rx=0 tx=0).stop 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 shutdown_connections 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f8a7006cce0 0x7f8a7006f1a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a881015a0 0x7f8a88100eb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 --2- 172.21.15.26:0/3037926818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a88103ed0 0x7f8a880ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 >> 172.21.15.26:0/3037926818 conn(0x7f8a880faf70 msgr2=0x7f8a880fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 shutdown_connections 2024-03-24T10:41:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:46.985+0000 7f8a8e2e5700 1 -- 172.21.15.26:0/3037926818 wait complete. 2024-03-24T10:41:47.686 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T10:41:47.911 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:47.950 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:47 smithi026 ceph-mon[29210]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-24T10:41:48.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:47 smithi031 ceph-mon[37345]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.674+0000 7feab5436700 1 -- 172.21.15.26:0/2218504141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 msgr2=0x7feab0101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.674+0000 7feab5436700 1 --2- 172.21.15.26:0/2218504141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0101040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009910 tx=0x7feaa4009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 -- 172.21.15.26:0/2218504141 shutdown_connections 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 --2- 172.21.15.26:0/2218504141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0101040 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 --2- 172.21.15.26:0/2218504141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab00ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 -- 172.21.15.26:0/2218504141 >> 172.21.15.26:0/2218504141 conn(0x7feab00faf80 msgr2=0x7feab00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 -- 172.21.15.26:0/2218504141 shutdown_connections 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 -- 172.21.15.26:0/2218504141 wait complete. 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.675+0000 7feab5436700 1 Processor -- start 2024-03-24T10:41:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feab5436700 1 -- start start 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feab5436700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab01953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feab5436700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feab5436700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0195f20 con 0x7feab0100be0 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feab5436700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0196060 con 0x7feab00ff9e0 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46476/0 (socket says 172.21.15.26:46476) 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 -- 172.21.15.26:0/1444925610 learned_addr learned my addr 172.21.15.26:0/1444925610 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaaeffd700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab01953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 -- 172.21.15.26:0/1444925610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 msgr2=0x7feab01953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab01953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaae7fc700 1 -- 172.21.15.26:0/1444925610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa40095f0 con 0x7feab0100be0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.676+0000 7feaaeffd700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab01953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7feaae7fc700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feaa400b800 tx=0x7feaa4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaa401d070 con 0x7feab0100be0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab0068f70 con 0x7feab0100be0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaa40052a0 con 0x7feab0100be0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab00694e0 con 0x7feab0100be0 2024-03-24T10:41:49.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.677+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaa4017400 con 0x7feab0100be0 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.678+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab004ea80 con 0x7feab0100be0 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.678+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7feaa4003780 con 0x7feab0100be0 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.679+0000 7fea97fff700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 0x7fea9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.679+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7feaa408d680 con 0x7feab0100be0 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.679+0000 7feaaeffd700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 0x7fea9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.680+0000 7feaaeffd700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 0x7fea9806f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feab0100a40 tx=0x7feaa0010040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.684+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feaa405b480 con 0x7feab0100be0 2024-03-24T10:41:49.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.902+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7feab018f370 con 0x7feab0100be0 2024-03-24T10:41:49.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.902+0000 7fea97fff700 1 -- 172.21.15.26:0/1444925610 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7feaa405b010 con 0x7feab0100be0 2024-03-24T10:41:49.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 msgr2=0x7fea9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 0x7fea9806f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feab0100a40 tx=0x7feaa0010040 comp rx=0 tx=0).stop 2024-03-24T10:41:49.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 msgr2=0x7feab0195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feaa400b800 tx=0x7feaa4005f00 comp rx=0 tx=0).stop 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 shutdown_connections 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fea9806cce0 0x7fea9806f1a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00ff9e0 0x7feab01953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 --2- 172.21.15.26:0/1444925610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feab0100be0 0x7feab0195900 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 >> 172.21.15.26:0/1444925610 conn(0x7feab00faf80 msgr2=0x7feab0104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 shutdown_connections 2024-03-24T10:41:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:49.905+0000 7feab5436700 1 -- 172.21.15.26:0/1444925610 wait complete. 2024-03-24T10:41:50.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:49 smithi031 ceph-mon[37345]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-03-24T10:41:50.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:49 smithi026 ceph-mon[29210]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-03-24T10:41:50.569 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-24T10:41:50.810 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:41:52.071 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:51 smithi026 ceph-mon[29210]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-03-24T10:41:52.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:51 smithi031 ceph-mon[37345]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-03-24T10:41:52.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 -- 172.21.15.26:0/1938689994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59801015c0 msgr2=0x7f59801039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 --2- 172.21.15.26:0/1938689994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59801015c0 0x7f59801039b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5974009910 tx=0x7f5974009c20 comp rx=0 tx=0).stop 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 -- 172.21.15.26:0/1938689994 shutdown_connections 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 --2- 172.21.15.26:0/1938689994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5980103ef0 0x7f59801062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 --2- 172.21.15.26:0/1938689994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59801015c0 0x7f59801039b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 -- 172.21.15.26:0/1938689994 >> 172.21.15.26:0/1938689994 conn(0x7f59800faf70 msgr2=0x7f59800fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 -- 172.21.15.26:0/1938689994 shutdown_connections 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.638+0000 7f5986fa4700 1 -- 172.21.15.26:0/1938689994 wait complete. 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 Processor -- start 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 -- start start 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59801015c0 0x7f5980198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5980199270 con 0x7f5980103ef0 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5986fa4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59801993b0 con 0x7f59801015c0 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f5984d40700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59801015c0 0x7f5980198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:52.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46492/0 (socket says 172.21.15.26:46492) 2024-03-24T10:41:52.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 -- 172.21.15.26:0/3658653322 learned_addr learned my addr 172.21.15.26:0/3658653322 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:41:52.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 -- 172.21.15.26:0/3658653322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59801015c0 msgr2=0x7f5980198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:52.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59801015c0 0x7f5980198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:52.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 -- 172.21.15.26:0/3658653322 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740095f0 con 0x7f5980103ef0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.639+0000 7f597ffff700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f596c00e940 tx=0x7f596c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.640+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c00cba0 con 0x7f5980103ef0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.640+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f596c00ee90 con 0x7f5980103ef0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.640+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598019de10 con 0x7f5980103ef0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.640+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c0055d0 con 0x7f5980103ef0 2024-03-24T10:41:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.640+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598019e3b0 con 0x7f5980103ef0 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.641+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f596c004750 con 0x7f5980103ef0 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.641+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598004ea80 con 0x7f5980103ef0 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.642+0000 7f597dffb700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 0x7f597006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.642+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f596c014070 con 0x7f5980103ef0 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.642+0000 7f5984d40700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 0x7f597006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.642+0000 7f5984d40700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 0x7f597006f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f597400b800 tx=0x7f5974009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.650+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f596c05a410 con 0x7f5980103ef0 2024-03-24T10:41:52.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:41:52.930+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f598019e730 con 0x7f597006ccf0 2024-03-24T10:41:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:53 smithi031 ceph-mon[37345]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:54.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:53 smithi031 ceph-mon[37345]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-24T10:41:54.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:53 smithi026 ceph-mon[29210]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:41:54.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:53 smithi026 ceph-mon[29210]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-24T10:41:56.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:56 smithi031 ceph-mon[37345]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-24T10:41:56.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:56 smithi026 ceph-mon[29210]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-03-24T10:41:58.180 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:57 smithi031 ceph-mon[37345]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-24T10:41:58.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:57 smithi026 ceph-mon[29210]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-24T10:41:59.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:58 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:41:59.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:58 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:42:00.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:41:59 smithi031 ceph-mon[37345]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-24T10:42:00.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:41:59 smithi026 ceph-mon[29210]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-24T10:42:02.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:01 smithi031 ceph-mon[37345]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:02.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:01 smithi026 ceph-mon[29210]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:04.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:03 smithi031 ceph-mon[37345]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:04.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:03 smithi026 ceph-mon[29210]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:06.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:06 smithi031 ceph-mon[37345]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:06.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:06 smithi026 ceph-mon[29210]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:07.783 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:07 smithi026 ceph-mon[29210]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:08.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:07 smithi031 ceph-mon[37345]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:10.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:09 smithi031 ceph-mon[37345]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:10.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:09 smithi026 ceph-mon[29210]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:12.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:11 smithi031 ceph-mon[37345]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:12.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:11 smithi026 ceph-mon[29210]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:13 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:42:14.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:13 smithi031 ceph-mon[37345]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:14.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:13 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:42:14.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:13 smithi026 ceph-mon[29210]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:42:16.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:16 smithi031 ceph-mon[37345]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-24T10:42:16.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:16 smithi026 ceph-mon[29210]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-24T10:42:18.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:17 smithi031 ceph-mon[37345]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-24T10:42:18.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:17 smithi026 ceph-mon[29210]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-24T10:42:20.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:19 smithi031 ceph-mon[37345]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-24T10:42:20.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:19 smithi026 ceph-mon[29210]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-24T10:42:22.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:21 smithi031 ceph-mon[37345]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-24T10:42:22.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:21 smithi026 ceph-mon[29210]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-24T10:42:23.850 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:23 smithi026 ceph-mon[29210]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-24T10:42:24.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:23 smithi031 ceph-mon[37345]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-24T10:42:25.276 INFO:tasks.workunit.client.1.smithi031.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T10:42:25.276 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.276 INFO:tasks.workunit.client.1.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T10:42:25.276 INFO:tasks.workunit.client.1.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: git switch -c 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:Or undo this operation with: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: git switch - 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr: 2024-03-24T10:42:25.277 INFO:tasks.workunit.client.1.smithi031.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T10:42:25.281 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-24T10:42:25.348 INFO:tasks.workunit.client.1.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T10:42:25.350 INFO:tasks.workunit.client.1.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T10:42:25.350 INFO:tasks.workunit.client.1.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T10:42:25.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.946+0000 7f597dffb700 1 -- 172.21.15.26:0/3658653322 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5974027070 con 0x7f597006ccf0 2024-03-24T10:42:25.947 INFO:teuthology.orchestra.run.smithi026.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.949+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 msgr2=0x7f597006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.949+0000 7f5986fa4700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 0x7f597006f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f597400b800 tx=0x7f5974009f70 comp rx=0 tx=0).stop 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.949+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 msgr2=0x7f5980198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.949+0000 7f5986fa4700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f596c00e940 tx=0x7f596c00ec50 comp rx=0 tx=0).stop 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 shutdown_connections 2024-03-24T10:42:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f597006ccf0 0x7f597006f1b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59801015c0 0x7f5980198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 --2- 172.21.15.26:0/3658653322 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5980103ef0 0x7f5980198c50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 >> 172.21.15.26:0/3658653322 conn(0x7f59800faf70 msgr2=0x7f59800ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 shutdown_connections 2024-03-24T10:42:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:25.950+0000 7f5986fa4700 1 -- 172.21.15.26:0/3658653322 wait complete. 2024-03-24T10:42:26.183 INFO:tasks.workunit.client.1.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T10:42:26.281 INFO:tasks.workunit.client.1.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T10:42:26.326 INFO:tasks.workunit.client.1.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T10:42:26.328 INFO:tasks.workunit.client.1.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T10:42:26.328 INFO:tasks.workunit.client.1.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T10:42:26.366 INFO:tasks.workunit.client.1.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T10:42:26.369 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T10:42:26.369 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-24T10:42:26.423 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-24T10:42:26.425 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T10:42:26.425 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-24T10:42:26.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:26 smithi031 ceph-mon[37345]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-24T10:42:26.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:26.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:26 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:26 smithi026 ceph-mon[29210]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-24T10:42:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:26.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:26 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:26.470 INFO:tasks.workunit.client.1.smithi031.stderr:+ mkdir -p fsstress 2024-03-24T10:42:26.472 INFO:tasks.workunit.client.1.smithi031.stderr:+ pushd fsstress 2024-03-24T10:42:26.472 INFO:tasks.workunit.client.1.smithi031.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T10:42:26.473 INFO:tasks.workunit.client.1.smithi031.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T10:42:26.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-24T10:42:26.965 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:42:27.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:27 smithi026 ceph-mon[29210]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:42:27.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:42:27.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:42:27.427 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:27 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:27.443 INFO:tasks.workunit.client.1.smithi031.stderr:+ tar xzf ltp-full.tgz 2024-03-24T10:42:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:27 smithi031 ceph-mon[37345]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:42:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:42:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:42:27.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:27 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:28.391 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:28 smithi026 ceph-mon[29210]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:42:28.392 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:28 smithi026 ceph-mon[29210]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-24T10:42:28.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:28 smithi031 ceph-mon[37345]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:42:28.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:28 smithi031 ceph-mon[37345]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.018+0000 7f1ed0e37700 1 -- 172.21.15.26:0/554485294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc100bd0 msgr2=0x7f1ecc101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.018+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/554485294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc101030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009910 tx=0x7f1ebc009c20 comp rx=0 tx=0).stop 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 -- 172.21.15.26:0/554485294 shutdown_connections 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/554485294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc101030 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/554485294 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 -- 172.21.15.26:0/554485294 >> 172.21.15.26:0/554485294 conn(0x7f1ecc0faf70 msgr2=0x7f1ecc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 -- 172.21.15.26:0/554485294 shutdown_connections 2024-03-24T10:42:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.019+0000 7f1ed0e37700 1 -- 172.21.15.26:0/554485294 wait complete. 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 Processor -- start 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 -- start start 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc071980 con 0x7f1ecc100bd0 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.020+0000 7f1ed0e37700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc071af0 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1ec9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1ec9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38982/0 (socket says 172.21.15.26:38982) 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1ec9d9b700 1 -- 172.21.15.26:0/3156097452 learned_addr learned my addr 172.21.15.26:0/3156097452 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1eca59c700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1eca59c700 1 -- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 msgr2=0x7f1ecc068e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.021+0000 7f1eca59c700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1eca59c700 1 -- 172.21.15.26:0/3156097452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0009520 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ec9d9b700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1eca59c700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc068950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e940 tx=0x7f1ec000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec000cb40 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ec000ee90 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec0018630 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0095f0 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.022+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc0721c0 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.024+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc06bf70 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.025+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f1ec000cca0 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.025+0000 7f1ebb7fe700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 0x7f1eb406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.026+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1ec0014070 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.026+0000 7f1ec9d9b700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 0x7f1eb406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.026+0000 7f1ec9d9b700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 0x7f1eb406f1b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc072c30 tx=0x7f1ebc005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:29.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.030+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ec00569e0 con 0x7f1ecc0ff9d0 2024-03-24T10:42:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.412+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ecc0611f0 con 0x7f1eb406ccf0 2024-03-24T10:42:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.413+0000 7f1ebb7fe700 1 -- 172.21.15.26:0/3156097452 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f1ecc0611f0 con 0x7f1eb406ccf0 2024-03-24T10:42:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.416+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 msgr2=0x7f1eb406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.416+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 0x7f1eb406f1b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc072c30 tx=0x7f1ebc005fb0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.416+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 msgr2=0x7f1ecc068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.416+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc068950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e940 tx=0x7f1ec000ec50 comp rx=0 tx=0).stop 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 shutdown_connections 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f1eb406ccf0 0x7f1eb406f1b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ecc0ff9d0 0x7f1ecc068950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 --2- 172.21.15.26:0/3156097452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ecc100bd0 0x7f1ecc068e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 >> 172.21.15.26:0/3156097452 conn(0x7f1ecc0faf70 msgr2=0x7f1ecc1041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.417+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 shutdown_connections 2024-03-24T10:42:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.418+0000 7f1ed0e37700 1 -- 172.21.15.26:0/3156097452 wait complete. 2024-03-24T10:42:29.429 INFO:teuthology.orchestra.run.smithi026.stdout:true 2024-03-24T10:42:29.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:29 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:42:29.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:29 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T10:42:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.583+0000 7fb5277fe700 1 -- 172.21.15.26:0/2356382062 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb51c00baf0 con 0x7fb5280fe990 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 -- 172.21.15.26:0/2356382062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 msgr2=0x7fb5280fedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 --2- 172.21.15.26:0/2356382062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb5280fedb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009910 tx=0x7fb51c009c20 comp rx=0 tx=0).stop 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 -- 172.21.15.26:0/2356382062 shutdown_connections 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 --2- 172.21.15.26:0/2356382062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb5280fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 --2- 172.21.15.26:0/2356382062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb5280fedb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.584+0000 7fb52f177700 1 -- 172.21.15.26:0/2356382062 >> 172.21.15.26:0/2356382062 conn(0x7fb5280fa2c0 msgr2=0x7fb5280fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.585+0000 7fb52f177700 1 -- 172.21.15.26:0/2356382062 shutdown_connections 2024-03-24T10:42:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.585+0000 7fb52f177700 1 -- 172.21.15.26:0/2356382062 wait complete. 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.585+0000 7fb52f177700 1 Processor -- start 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.585+0000 7fb52f177700 1 -- start start 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52f177700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52f177700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb52819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52f177700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52819d600 con 0x7fb5280ffad0 2024-03-24T10:42:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52f177700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52819d740 con 0x7fb5280fe990 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52cf13700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52cf13700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.26:51180/0 (socket says 172.21.15.26:51180) 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb52cf13700 1 -- 172.21.15.26:0/1274815563 learned_addr learned my addr 172.21.15.26:0/1274815563 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.586+0000 7fb527fff700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb52819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb52cf13700 1 -- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 msgr2=0x7fb52819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb52cf13700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb52819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb52cf13700 1 -- 172.21.15.26:0/1274815563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb514009520 con 0x7fb5280fe990 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb527fff700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb52819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb52cf13700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005f30 tx=0x7fb51c005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb51c01d070 con 0x7fb5280fe990 2024-03-24T10:42:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb51c0052a0 con 0x7fb5280fe990 2024-03-24T10:42:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.587+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb51c017400 con 0x7fb5280fe990 2024-03-24T10:42:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.588+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c0095f0 con 0x7fb5280fe990 2024-03-24T10:42:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.588+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5281a2520 con 0x7fb5280fe990 2024-03-24T10:42:29.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.590+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb528196c60 con 0x7fb5280fe990 2024-03-24T10:42:29.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.591+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fb51c003680 con 0x7fb5280fe990 2024-03-24T10:42:29.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.591+0000 7fb525ffb700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 0x7fb51806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:29.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.591+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb51c08d4b0 con 0x7fb5280fe990 2024-03-24T10:42:29.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.591+0000 7fb527fff700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 0x7fb51806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:29.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.592+0000 7fb527fff700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 0x7fb51806f1a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb514009fd0 tx=0x7fb514009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:29.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:29.597+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb51c05b3d0 con 0x7fb5280fe990 2024-03-24T10:42:30.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.042+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb52802d0d0 con 0x7fb51806cce0 2024-03-24T10:42:30.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.045+0000 7fb525ffb700 1 -- 172.21.15.26:0/1274815563 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fb52802d0d0 con 0x7fb51806cce0 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.048+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 msgr2=0x7fb51806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.048+0000 7fb52f177700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 0x7fb51806f1a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb514009fd0 tx=0x7fb514009260 comp rx=0 tx=0).stop 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 msgr2=0x7fb52819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005f30 tx=0x7fb51c005960 comp rx=0 tx=0).stop 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 shutdown_connections 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7fb51806cce0 0x7fb51806f1a0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb514009fd0 tx=0x7fb514009260 comp rx=0 tx=0).stop 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5280fe990 0x7fb52819caa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 --2- 172.21.15.26:0/1274815563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5280ffad0 0x7fb52819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 >> 172.21.15.26:0/1274815563 conn(0x7fb5280fa2c0 msgr2=0x7fb5281030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:30.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.049+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 shutdown_connections 2024-03-24T10:42:30.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.050+0000 7fb52f177700 1 -- 172.21.15.26:0/1274815563 wait complete. 2024-03-24T10:42:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 -- 172.21.15.26:0/2748856733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 msgr2=0x7f04c80a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 --2- 172.21.15.26:0/2748856733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 0x7f04c80a4e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009910 tx=0x7f04cc009c20 comp rx=0 tx=0).stop 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 -- 172.21.15.26:0/2748856733 shutdown_connections 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 --2- 172.21.15.26:0/2748856733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c80a5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 --2- 172.21.15.26:0/2748856733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 0x7f04c80a4e20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.221+0000 7f04d5857700 1 -- 172.21.15.26:0/2748856733 >> 172.21.15.26:0/2748856733 conn(0x7f04c809fec0 msgr2=0x7f04c80a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.222+0000 7f04d5857700 1 -- 172.21.15.26:0/2748856733 shutdown_connections 2024-03-24T10:42:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.222+0000 7f04d5857700 1 -- 172.21.15.26:0/2748856733 wait complete. 2024-03-24T10:42:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.222+0000 7f04d5857700 1 Processor -- start 2024-03-24T10:42:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.222+0000 7f04d5857700 1 -- start start 2024-03-24T10:42:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04d5857700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 0x7f04c8142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04d5857700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04d5857700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c81436f0 con 0x7f04c80a4a00 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04d5857700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8143830 con 0x7f04c80a5b40 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04c7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04c7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.26:51192/0 (socket says 172.21.15.26:51192) 2024-03-24T10:42:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.223+0000 7f04c7fff700 1 -- 172.21.15.26:0/2447348527 learned_addr learned my addr 172.21.15.26:0/2447348527 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c7fff700 1 -- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 msgr2=0x7f04c8142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c7fff700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 0x7f04c8142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c7fff700 1 -- 172.21.15.26:0/2447348527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0095f0 con 0x7f04c80a5b40 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c7fff700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04c000d8d0 tx=0x7f04c000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c0009940 con 0x7f04c80a5b40 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04c0010460 con 0x7f04c80a5b40 2024-03-24T10:42:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.224+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c8148240 con 0x7f04c80a5b40 2024-03-24T10:42:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.225+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c000f5f0 con 0x7f04c80a5b40 2024-03-24T10:42:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.225+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c8148810 con 0x7f04c80a5b40 2024-03-24T10:42:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.226+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c8004ce0 con 0x7f04c80a5b40 2024-03-24T10:42:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.227+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f04c00105d0 con 0x7f04c80a5b40 2024-03-24T10:42:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.228+0000 7f04c5ffb700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 0x7f04b806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.228+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f04c008c2a0 con 0x7f04c80a5b40 2024-03-24T10:42:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.229+0000 7f04d4855700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 0x7f04b806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.229+0000 7f04d4855700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 0x7f04b806f1f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04cc005090 tx=0x7f04cc005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.232+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f04c005a080 con 0x7f04c80a5b40 2024-03-24T10:42:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:30 smithi026 ceph-mon[29210]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:30 smithi026 ceph-mon[29210]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-24T10:42:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:30.612 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:30 smithi026 ceph-mon[29210]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:30.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.670+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c813b530 con 0x7f04b806cd30 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.smithi026 smithi026 *:9093,9094 running (5m) 56s ago 6m 25.8M - 0.25.0 c8568f914cd2 3f177e8f816c 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-exporter.smithi026 smithi026 running (7m) 56s ago 7m 7419k - 18.2.2-952-g738c2fd6 68aff9aff3b2 044d4f7bec30 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-exporter.smithi031 smithi031 running (6m) 61s ago 6m 7184k - 18.2.2-952-g738c2fd6 68aff9aff3b2 08932fd65525 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi026 smithi026 running (7m) 56s ago 7m 6660k - 18.2.2-952-g738c2fd6 68aff9aff3b2 04efe48df2e6 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:crash.smithi031 smithi031 running (6m) 61s ago 6m 6660k - 18.2.2-952-g738c2fd6 68aff9aff3b2 c6b079e4b35c 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:grafana.smithi026 smithi026 *:3000 running (5m) 56s ago 6m 84.2M - 9.4.7 954c08fa6188 4696b23e1eeb 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.dfxoad smithi026 running (81s) 56s ago 80s 11.3M - 18.2.2-952-g738c2fd6 68aff9aff3b2 3e216c2e610e 2024-03-24T10:42:30.692 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi026.ryhyqm smithi026 running (74s) 56s ago 74s 14.7M - 18.2.2-952-g738c2fd6 68aff9aff3b2 1f37ea31990f 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi031.cgovmq smithi031 running (78s) 61s ago 77s 15.4M - 18.2.2-952-g738c2fd6 68aff9aff3b2 35868b630382 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mds.cephfs.smithi031.vjdtgj smithi031 running (72s) 61s ago 71s 14.6M - 18.2.2-952-g738c2fd6 68aff9aff3b2 30eada913d84 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi026.bbjcru smithi026 *:9283,8765,8443 running (8m) 56s ago 8m 497M - 18.2.2-952-g738c2fd6 68aff9aff3b2 181e5b6ed806 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.smithi031.ucwncv smithi031 *:8443,9283,8765 running (5m) 61s ago 5m 450M - 18.2.2-952-g738c2fd6 68aff9aff3b2 ff3fb3d02a0e 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi026 smithi026 running (8m) 56s ago 8m 47.9M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 dfd13627777b 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:mon.smithi031 smithi031 running (5m) 61s ago 5m 35.2M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 72bc55976cf6 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi026 smithi026 *:9100 running (7m) 56s ago 7m 16.3M - 1.5.0 0da6a335fe13 850149efce92 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter.smithi031 smithi031 *:9100 running (5m) 61s ago 5m 15.4M - 1.5.0 0da6a335fe13 ef178edadca0 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (5m) 56s ago 5m 50.3M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 6dd317107551 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (4m) 56s ago 4m 46.3M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 8bf2a3e7910d 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (4m) 56s ago 4m 46.6M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 d9c7f3ffbeae 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi031 running (3m) 61s ago 3m 45.9M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 afb16aab0936 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi031 running (3m) 61s ago 3m 45.9M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 6b4b79fce473 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi031 running (2m) 61s ago 2m 45.8M 4972M 18.2.2-952-g738c2fd6 68aff9aff3b2 1851387fda3d 2024-03-24T10:42:30.693 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.smithi026 smithi026 *:9095 running (5m) 56s ago 6m 54.3M - 2.43.0 a07b618ecd1d fb76b5ad356d 2024-03-24T10:42:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.690+0000 7f04c5ffb700 1 -- 172.21.15.26:0/2447348527 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3792 (secure 0 0 0) 0x7f04c813b530 con 0x7f04b806cd30 2024-03-24T10:42:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 msgr2=0x7f04b806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 0x7f04b806f1f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04cc005090 tx=0x7f04cc005fb0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 msgr2=0x7f04c81430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04c000d8d0 tx=0x7f04c000dc90 comp rx=0 tx=0).stop 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 shutdown_connections 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f04b806cd30 0x7f04b806f1f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04c80a4a00 0x7f04c8142b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 --2- 172.21.15.26:0/2447348527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04c80a5b40 0x7f04c81430d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 >> 172.21.15.26:0/2447348527 conn(0x7f04c809fec0 msgr2=0x7f04c80a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 shutdown_connections 2024-03-24T10:42:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.694+0000 7f04d5857700 1 -- 172.21.15.26:0/2447348527 wait complete. 2024-03-24T10:42:30.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:30 smithi031 ceph-mon[37345]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:30.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:30 smithi031 ceph-mon[37345]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-24T10:42:30.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:30.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:30.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:30 smithi031 ceph-mon[37345]: from='mgr.14215 172.21.15.26:0/11932360' entity='mgr.smithi026.bbjcru' 2024-03-24T10:42:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.868+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1888554217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80ff1c0 msgr2=0x7f32f81061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.869+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1888554217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80ff1c0 0x7f32f81061d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f32f000b0a0 tx=0x7f32f000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.870+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1888554217 shutdown_connections 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.870+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1888554217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80ff1c0 0x7f32f81061d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.870+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1888554217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f80fe860 0x7f32f80fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.870+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1888554217 >> 172.21.15.26:0/1888554217 conn(0x7f32f80fa2c0 msgr2=0x7f32f80fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.871+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1888554217 shutdown_connections 2024-03-24T10:42:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.871+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1888554217 wait complete. 2024-03-24T10:42:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 Processor -- start 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 -- start start 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f80ff1c0 0x7f32f8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8199280 con 0x7f32f80ff1c0 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32fe6d8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f81993c0 con 0x7f32f80fe860 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.26:51198/0 (socket says 172.21.15.26:51198) 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.872+0000 7f32f7fff700 1 -- 172.21.15.26:0/1779529239 learned_addr learned my addr 172.21.15.26:0/1779529239 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.873+0000 7f32f7fff700 1 -- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f80ff1c0 msgr2=0x7f32f8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.873+0000 7f32f7fff700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f80ff1c0 0x7f32f8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.873+0000 7f32f7fff700 1 -- 172.21.15.26:0/1779529239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0009d00 con 0x7f32f80fe860 2024-03-24T10:42:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.873+0000 7f32f7fff700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f32e000e920 tx=0x7f32e000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.874+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e000cba0 con 0x7f32f80fe860 2024-03-24T10:42:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.874+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f819de20 con 0x7f32f80fe860 2024-03-24T10:42:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.874+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f819e390 con 0x7f32f80fe860 2024-03-24T10:42:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.874+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32e000cd00 con 0x7f32f80fe860 2024-03-24T10:42:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.874+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e000cba0 con 0x7f32f80fe860 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.876+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8005320 con 0x7f32f80fe860 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.877+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f32e0010af0 con 0x7f32f80fe860 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.877+0000 7f32f57fa700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 0x7f32e406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.877+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f32e0014070 con 0x7f32f80fe860 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.878+0000 7f32f77fe700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 0x7f32e406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.878+0000 7f32f77fe700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 0x7f32e406f260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f32f0015040 tx=0x7f32f00077e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:30.883+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32e0056e60 con 0x7f32f80fe860 2024-03-24T10:42:31.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.468+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f32d8005cc0 con 0x7f32f80fe860 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": { 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": { 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": { 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 6 2024-03-24T10:42:31.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": { 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 4 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: "overall": { 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 14 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-24T10:42:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.469+0000 7f32f57fa700 1 -- 172.21.15.26:0/1779529239 <== mon.1 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f32e0057000 con 0x7f32f80fe860 2024-03-24T10:42:31.471 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: Upgrade: Target is version 19.0.0-1855-g21b3bd67 (unknown) 2024-03-24T10:42:31.472 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919'] 2024-03-24T10:42:31.472 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: Upgrade: Need to upgrade myself (mgr.smithi026.bbjcru) 2024-03-24T10:42:31.472 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:31.472 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 on smithi031 2024-03-24T10:42:31.472 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:31 smithi026 ceph-mon[29210]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 msgr2=0x7f32e406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 0x7f32e406f260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f32f0015040 tx=0x7f32f00077e0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 msgr2=0x7f32f8198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f32e000e920 tx=0x7f32e000ece0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 shutdown_connections 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7f32e406cda0 0x7f32e406f260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f80fe860 0x7f32f8198720 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 --2- 172.21.15.26:0/1779529239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f80ff1c0 0x7f32f8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 >> 172.21.15.26:0/1779529239 conn(0x7f32f80fa2c0 msgr2=0x7f32f8103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 shutdown_connections 2024-03-24T10:42:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.473+0000 7f32fe6d8700 1 -- 172.21.15.26:0/1779529239 wait complete. 2024-03-24T10:42:31.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: Upgrade: Target is version 19.0.0-1855-g21b3bd67 (unknown) 2024-03-24T10:42:31.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919'] 2024-03-24T10:42:31.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: Upgrade: Need to upgrade myself (mgr.smithi026.bbjcru) 2024-03-24T10:42:31.649 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:31.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 on smithi031 2024-03-24T10:42:31.650 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:31 smithi031 ceph-mon[37345]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffab5ffb700 1 -- 172.21.15.26:0/2186879887 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaa000baf0 con 0x7ffab00943e0 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 -- 172.21.15.26:0/2186879887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab00943e0 msgr2=0x7ffab0096810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 --2- 172.21.15.26:0/2186879887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab00943e0 0x7ffab0096810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009910 tx=0x7ffaa0009c20 comp rx=0 tx=0).stop 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 -- 172.21.15.26:0/2186879887 shutdown_connections 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 --2- 172.21.15.26:0/2186879887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 0x7ffab00991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 --2- 172.21.15.26:0/2186879887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab00943e0 0x7ffab0096810 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009910 tx=0x7ffaa0009c20 comp rx=0 tx=0).stop 2024-03-24T10:42:31.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 -- 172.21.15.26:0/2186879887 >> 172.21.15.26:0/2186879887 conn(0x7ffab0090020 msgr2=0x7ffab0092440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:31.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.653+0000 7ffabd504700 1 -- 172.21.15.26:0/2186879887 shutdown_connections 2024-03-24T10:42:31.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.654+0000 7ffabd504700 1 -- 172.21.15.26:0/2186879887 wait complete. 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.654+0000 7ffabd504700 1 Processor -- start 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.654+0000 7ffabd504700 1 -- start start 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffabd504700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 0x7ffab0131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffabd504700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffabd504700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab0132750 con 0x7ffab0096d50 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffabd504700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab01328c0 con 0x7ffab01322d0 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffab67fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffab67fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.26:51206/0 (socket says 172.21.15.26:51206) 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffab67fc700 1 -- 172.21.15.26:0/85895798 learned_addr learned my addr 172.21.15.26:0/85895798 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-03-24T10:42:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.655+0000 7ffab6ffd700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 0x7ffab0131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffab67fc700 1 -- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 msgr2=0x7ffab0131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffab67fc700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 0x7ffab0131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffab67fc700 1 -- 172.21.15.26:0/85895798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa00095f0 con 0x7ffab01322d0 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffab67fc700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800e920 tx=0x7ffaa800ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaa800c9d0 con 0x7ffab01322d0 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffaa8010430 con 0x7ffab01322d0 2024-03-24T10:42:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaa8018630 con 0x7ffab01322d0 2024-03-24T10:42:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab0137850 con 0x7ffab01322d0 2024-03-24T10:42:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.656+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab0137df0 con 0x7ffab01322d0 2024-03-24T10:42:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.658+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab00055b0 con 0x7ffab01322d0 2024-03-24T10:42:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.659+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ffaa80105a0 con 0x7ffab01322d0 2024-03-24T10:42:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.659+0000 7ffaaffff700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 0x7ffaa406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.660+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffaa8014070 con 0x7ffab01322d0 2024-03-24T10:42:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.660+0000 7ffab6ffd700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 0x7ffaa406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.661+0000 7ffab6ffd700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 0x7ffaa406f1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0000c00 tx=0x7ffaa0005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:31.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:31.665+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mon.1 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffaa80529f0 con 0x7ffab01322d0 2024-03-24T10:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.116+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 --> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffab01380a0 con 0x7ffaa406cd30 2024-03-24T10:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.125+0000 7ffaaffff700 1 -- 172.21.15.26:0/85895798 <== mgr.14215 v2:172.21.15.26:6800/3570278484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ffab01380a0 con 0x7ffaa406cd30 2024-03-24T10:42:32.126 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-24T10:42:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919", 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "in_progress": true, 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "services_complete": [], 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "progress": "0/2 daemons upgraded", 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 image on host smithi031", 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "is_paused": false 2024-03-24T10:42:32.127 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.129+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 msgr2=0x7ffaa406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.129+0000 7ffabd504700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 0x7ffaa406f1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0000c00 tx=0x7ffaa0005fb0 comp rx=0 tx=0).stop 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.129+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 msgr2=0x7ffab01372b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.129+0000 7ffabd504700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800e920 tx=0x7ffaa800ec30 comp rx=0 tx=0).stop 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 shutdown_connections 2024-03-24T10:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:6800/3570278484,v1:172.21.15.26:6801/3570278484] conn(0x7ffaa406cd30 0x7ffaa406f1f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab0096d50 0x7ffab0131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 --2- 172.21.15.26:0/85895798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffab01322d0 0x7ffab01372b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 >> 172.21.15.26:0/85895798 conn(0x7ffab0090020 msgr2=0x7ffab0092410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.130+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 shutdown_connections 2024-03-24T10:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:42:32.131+0000 7ffabd504700 1 -- 172.21.15.26:0/85895798 wait complete. 2024-03-24T10:42:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:32 smithi026 ceph-mon[29210]: from='client.? 172.21.15.26:0/1779529239' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T10:42:32.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:32 smithi026 ceph-mon[29210]: pgmap v224: 65 pgs: 65 active+clean; 4.6 MiB data, 173 MiB used, 536 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-03-24T10:42:32.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:32 smithi031 ceph-mon[37345]: from='client.? 172.21.15.26:0/1779529239' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T10:42:32.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:32 smithi031 ceph-mon[37345]: pgmap v224: 65 pgs: 65 active+clean; 4.6 MiB data, 173 MiB used, 536 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-03-24T10:42:33.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:33 smithi031 ceph-mon[37345]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:33.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:33 smithi026 ceph-mon[29210]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:42:34.700 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:34 smithi031 ceph-mon[37345]: pgmap v225: 65 pgs: 65 active+clean; 17 MiB data, 194 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 46 op/s 2024-03-24T10:42:34.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:34 smithi026 ceph-mon[29210]: pgmap v225: 65 pgs: 65 active+clean; 17 MiB data, 194 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 46 op/s 2024-03-24T10:42:35.000 INFO:tasks.workunit.client.0.smithi026.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: git switch -c 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr:Or undo this operation with: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.001 INFO:tasks.workunit.client.0.smithi026.stderr: git switch - 2024-03-24T10:42:35.002 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.002 INFO:tasks.workunit.client.0.smithi026.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T10:42:35.002 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T10:42:35.002 INFO:tasks.workunit.client.0.smithi026.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T10:42:35.009 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T10:42:35.040 INFO:tasks.workunit.client.0.smithi026.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T10:42:35.043 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T10:42:35.043 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T10:42:36.450 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:36 smithi031 ceph-mon[37345]: pgmap v226: 65 pgs: 65 active+clean; 29 MiB data, 256 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s wr, 180 op/s 2024-03-24T10:42:36.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:36 smithi026 ceph-mon[29210]: pgmap v226: 65 pgs: 65 active+clean; 29 MiB data, 256 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s wr, 180 op/s 2024-03-24T10:42:38.199 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:37 smithi031 ceph-mon[37345]: pgmap v227: 65 pgs: 65 active+clean; 39 MiB data, 293 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 284 op/s 2024-03-24T10:42:38.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:37 smithi026 ceph-mon[29210]: pgmap v227: 65 pgs: 65 active+clean; 39 MiB data, 293 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s wr, 284 op/s 2024-03-24T10:42:38.671 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T10:42:38.989 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T10:42:39.575 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T10:42:39.576 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T10:42:39.576 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T10:42:39.740 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T10:42:39.743 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:42:39.743 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T10:42:39.799 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-24T10:42:39.800 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T10:42:39.800 DEBUG:teuthology.orchestra.run.smithi026:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-24T10:42:39.859 INFO:tasks.workunit.client.0.smithi026.stderr:+ mkdir -p fsstress 2024-03-24T10:42:39.861 INFO:tasks.workunit.client.0.smithi026.stderr:+ pushd fsstress 2024-03-24T10:42:39.861 INFO:tasks.workunit.client.0.smithi026.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-24T10:42:39.861 INFO:tasks.workunit.client.0.smithi026.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T10:42:40.200 INFO:journalctl@ceph.mon.smithi031.smithi031.stdout:Mar 24 10:42:39 smithi031 ceph-mon[37345]: pgmap v228: 65 pgs: 65 active+clean; 43 MiB data, 308 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 285 op/s 2024-03-24T10:42:40.200 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:39 smithi026 ceph-mon[29210]: pgmap v228: 65 pgs: 65 active+clean; 43 MiB data, 308 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 285 op/s 2024-03-24T10:42:41.188 INFO:tasks.workunit.client.0.smithi026.stderr:+ tar xzf ltp-full.tgz 2024-03-24T10:42:41.358 INFO:tasks.workunit.client.1.smithi031.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-24T10:42:41.358 INFO:tasks.workunit.client.1.smithi031.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T10:42:41.358 INFO:tasks.workunit.client.1.smithi031.stderr:+ make 2024-03-24T10:42:41.778 INFO:tasks.workunit.client.1.smithi031.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-24T10:42:42.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:42:41 smithi026 ceph-mon[29210]: pgmap v229: 65 pgs: 65 active+clean; 55 MiB data, 348 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 294 op/s 2024-03-24T10:43:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:05.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:05.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:05.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:05.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:05.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:05.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:06.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:06.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:06.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:06.753 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:06.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:06.753 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:06.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:06.753 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:06.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:07.112 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:06.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:07.112 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:06.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:07.112 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:06.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:07.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:07.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:07.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:07.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:07.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:07.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:07.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:07.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:07.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:08.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:08.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:08.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:08.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:08.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:08.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:08.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:08.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:08.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:09.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:09.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:09.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:09.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:09.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:09.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:09.694+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:09.694+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:09.694+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:10.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:10.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:10.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:10.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:10.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:10.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:11.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:11.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:11.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:11.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:11.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:11.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:11.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:11.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:11.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:12.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:12.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:12.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:12.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:12.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:12.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:12.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:12.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:12.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:13.827 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:13.691+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:13.827 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:13.691+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:13.827 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:13.691+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:14.124 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:13.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.124 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:13.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.124 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:13.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:14.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:14.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:14.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:14.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:14.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:14.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:14.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:14.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:14.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:15.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:15.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:15.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:15.790+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:15.790+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:15.790+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:15.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:15.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:15.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:15.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:15.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:16.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:16.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:16.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:16.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:16.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:16.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:16.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:16.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:16.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:16.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:16.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:16.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:17.155+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:17.155+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:17.155+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:17.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:17.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:17.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:17.791+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:17.791+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:17.791+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:18.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:18.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:18.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:18.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:18.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:18.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:18.744+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:18.744+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:18.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:18.744+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:19.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:19.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:19.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:19.649+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:19.649+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:19.649+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:19.727+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:19.727+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:19.727+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:20.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:20.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:20.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:20.633+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:20.633+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:20.633+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:20.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:20.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:20.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:21.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:21.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:21.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:21.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:21.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:21.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:21.700+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:21.700+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:21.700+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:22.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:22.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:22.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:22.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:22.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:22.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:22.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:22.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:22.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:23.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:23.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:23.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:23.661+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:23.661+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:23.661+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:23.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:23.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:23.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:24.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:24.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:24.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:24.654+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:24.654+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:24.654+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:24.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:24.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:24.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:25.306 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:25.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:25.306 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:25.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:25.306 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:25.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:25.668+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:25.668+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:25.668+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:25.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:25.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:25.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:26.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:26.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:26.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:26.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:26.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:26.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:26.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:26.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:26.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:27.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:27.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:27.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:27.709+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:27.709+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:27.709+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:27.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:27.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:27.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:28.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:28.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:28.028+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:28.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:28.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:28.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:28.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:28.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:28.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:29.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:29.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:29.287 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:29.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:29.287 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:29.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:29.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:29.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:29.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:29.710+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:29.710+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:29.710+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:30.305 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:30.305 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:30.306 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:30.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:30.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:30.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:30.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:30.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:30.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:31.758 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:31.452+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:31.758 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:31.452+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:31.758 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:31.452+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:31.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:31.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:31.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:31.758+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:31.758+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:31.758+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:32.732 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:32.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:32.733 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:32.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:32.733 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:32.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:32.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:32.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:32.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:32.732+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:32.732+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:33.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:32.732+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:33.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:33.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:33.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:33.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:33.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:33.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:33.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:33.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:33.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:33.746+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:33.746+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:34.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:33.746+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:34.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:34.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:34.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:35.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:34.981+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:35.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:34.981+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:35.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:34.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:35.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:34.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:34.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:34.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:35.755+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:35.755+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:35.755+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:35.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:35.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:35.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:35.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:35.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:35.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:36.773 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:36.497+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:36.773 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:36.497+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:36.773 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:36.497+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:36.773+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:36.773+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:36.773+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:37.800 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:37.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:37.800 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:37.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:37.800 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:37.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:37.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:37.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:37.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:37.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:38.757 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:38.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:38.757 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:38.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:38.757 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:38.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:39.008 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:39.008 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:39.008 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:39.381 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:39.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:39.381 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:39.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:39.381 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:39.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:39.513+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:39.513+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:39.513+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:40.013 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:39.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:40.014 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:39.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:40.014 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:39.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:40.305 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:40.013+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:40.305 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:40.013+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:40.306 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:40.013+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:40.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:40.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:40.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:41.047 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:40.794+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:41.047 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:40.794+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:41.047 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:40.794+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:41.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:41.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:41.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:41.836 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:41.510+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:41.836 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:41.510+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:41.836 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:41.510+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:42.074+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:42.074+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:42.074+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:41.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:41.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:42.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:41.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:42.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:42.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:42.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:42.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:42.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:42.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:43.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:43.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:43.076+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:42.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:42.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:43.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:42.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:43.809 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:43.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:43.809 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:43.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:43.809 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:43.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:43.808+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:43.808+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:43.808+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:44.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:44.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:44.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:45.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:44.821+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:45.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:44.821+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:45.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:44.821+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:45.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:45.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:45.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:44.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:45.849 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:45.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:45.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:45.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:45.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:45.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:45.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:45.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:45.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:45.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:45.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:45.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:46.866 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:46.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:46.866 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:46.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:46.866 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:46.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:46.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:46.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:46.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:47.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:47.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:47.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:47.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:47.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:47.835+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:47.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:47.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:47.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.015+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.015+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.015+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:48.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:48.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:48.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:48.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:48.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:48.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:48.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:49.862 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:49.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:49.862 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:49.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:49.862 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:49.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:49.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:49.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:49.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:49.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:49.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:49.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:50.923 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:50.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:50.924 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:50.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:50.924 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:50.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:50.924 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:50.856+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:50.924 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:50.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:50.924 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:50.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:50.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:50.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:50.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:51.899 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:51.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:51.899 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:51.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:51.899 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:51.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:51.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:51.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:51.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:51.898+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:51.898+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:51.898+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:52.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:52.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:52.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:52.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:52.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:52.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:52.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:52.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:53.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:53.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:53.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:53.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:53.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:53.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:53.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:53.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:53.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:53.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:53.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:54.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:54.753+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:54.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:54.753+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:54.858 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:54.753+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:55.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:55.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:55.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:54.975+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:55.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:54.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:55.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:54.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:55.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:54.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:55.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:55.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:55.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:55.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:55.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:55.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:55.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:55.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:55.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:56.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:56.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:56.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:56.789+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:56.789+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:56.789+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:57.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:57.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:57.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:57.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:57.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:57.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:57.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:58.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:58.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:43:58.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:43:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:58.836+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:58.836+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:59.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:58.836+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:43:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:58.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:58.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:58.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:59.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:59.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:59.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:59.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:43:59.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:43:59.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:00.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:00.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:00.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:00.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:59.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:00.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:59.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:00.196 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:43:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:43:59.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:00.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:00.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:00.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:00.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:00.896+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:00.896+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:00.896+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:01.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:01.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:01.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:02.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:02.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:02.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:01.923+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:01.923+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:01.923+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:03.032 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:02.903+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:03.033 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:02.903+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:03.033 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:02.903+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:03.033 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:02.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:03.033 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:02.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:03.033 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:02.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:03.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:03.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:03.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:03.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:03.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:03.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:03.868+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:03.868+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:03.868+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:04.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:04.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:04.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:04.909 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:04.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:04.910 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:04.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:04.910 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:04.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:05.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:05.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:05.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:04.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:04.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:05.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:04.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:06.072 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:05.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:06.072 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:05.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:06.072 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:05.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:06.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:05.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:06.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:05.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:06.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:05.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:06.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:06.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:06.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:07.042 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:06.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:07.042 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:06.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:07.043 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:06.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:07.043 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:06.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:07.043 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:06.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:07.043 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:06.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:07.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:07.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:07.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:08.046 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:07.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:08.046 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:07.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:08.046 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:07.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:08.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:07.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:08.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:07.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:08.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:07.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:08.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:08.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:08.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:09.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:09.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:09.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:08.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:08.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:08.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:08.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:08.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:08.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:09.948 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:09.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:09.948 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:09.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:09.948 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:09.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:10.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:10.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:10.046+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:09.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:09.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:09.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:11.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:11.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:11.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:10.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:10.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:10.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:10.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:10.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:10.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:11.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:11.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:11.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:12.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:12.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:12.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:11.857+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:11.857+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:11.857+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:12.883+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:12.883+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:12.883+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:12.839+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:12.839+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:12.839+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:14.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:14.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:13.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:13.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:13.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:13.858+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:13.873+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:13.873+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.190 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:13.873+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:14.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:14.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:14.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:14.844+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:14.844+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:14.977 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:14.844+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:15.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:14.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:15.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:14.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:15.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:14.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:15.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:15.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:15.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:15.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:15.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:15.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:15.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:15.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:15.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:16.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:16.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:16.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:17.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:16.915+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:16.915+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:16.915+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:16.915+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:16.915+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:16.915+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:17.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:17.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:17.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:17.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:17.866+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:17.866+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:17.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:17.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:17.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:18.913+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:18.913+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:18.913+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:18.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:18.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:18.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:18.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:18.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:18.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:19.917+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:19.917+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:19.917+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:19.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:19.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:20.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:19.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:20.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:19.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:20.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:19.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:20.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:19.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:20.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:20.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:20.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:20.966+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:20.966+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:20.966+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:20.999+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:20.999+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:20.999+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:21.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:21.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:21.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:21.878+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:21.878+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:21.878+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:22.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:22.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:22.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:22.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:22.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:22.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:22.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:23.997+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:23.997+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:23.997+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:23.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:23.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:23.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:23.956+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:23.956+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:23.956+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:25.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:24.950+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:25.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:24.950+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:25.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:24.950+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:24.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:24.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:24.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:24.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:24.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:25.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:24.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:25.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:25.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:25.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:26.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:25.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:26.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:25.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:26.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:25.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:26.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:25.943+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:26.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:25.943+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:26.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:25.944+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:26.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:26.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:26.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:26.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:26.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:26.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:26.942+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:26.942+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:26.942+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:27.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:27.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:27.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:27.950+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:27.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:27.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:27.976+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:27.976+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:27.976+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:28.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:28.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:28.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:28.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:28.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:28.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:28.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:28.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:29.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:28.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:29.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:29.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:29.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:29.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:29.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:29.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:29.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:30.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:29.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:29.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:30.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:30.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:30.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:30.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:30.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:30.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:31.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:30.916+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:31.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:30.916+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:31.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:30.916+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:31.920+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:31.920+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:31.920+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:31.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:31.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:31.920+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:31.864+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:31.864+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:31.864+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:32.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:32.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:32.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:32.908+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:32.908+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:32.908+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:32.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:32.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:32.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:33.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:33.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:33.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:33.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:33.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:33.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:33.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:33.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:33.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:34.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:35.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:34.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:35.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:34.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:34.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:34.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:34.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:34.918+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:34.918+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:34.918+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:35.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:35.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:35.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:35.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:35.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:35.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:35.950+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:35.950+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:35.950+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:37.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:36.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:37.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:36.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:37.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:36.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:36.843+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:36.843+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:36.843+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:36.965+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:36.966+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:36.966+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:37.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:37.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:37.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:38.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:37.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:38.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:37.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:38.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:37.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:38.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:39.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:38.832+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:39.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:38.832+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:39.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:38.832+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:40.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:40.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:40.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:39.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:39.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:39.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:39.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:39.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:40.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:39.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.971+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.971+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:40.971+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:40.870+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:40.870+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:40.870+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:41.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.020+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.020+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.020+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:41.847+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:41.847+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:41.848+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.993+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.993+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:42.993+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:43.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:42.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:42.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:42.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:44.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:44.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:44.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:43.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:43.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:43.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:43.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:45.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:45.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:45.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:44.796+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:44.796+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:44.796+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:44.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:44.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:44.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:45.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:45.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:45.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:46.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:46.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:46.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:45.771+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:45.771+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:45.771+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:46.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:46.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:46.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:47.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:46.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:47.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:46.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:47.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:46.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:47.901+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:47.901+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:47.901+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:47.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:47.741+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:47.741+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:47.741+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:48.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:48.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:48.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:49.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:49.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:49.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:48.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:48.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:48.781+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:50.191 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:50.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:50.191 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:50.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:50.191 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:50.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:49.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:49.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:49.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:49.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:49.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:50.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:49.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:44:50.190+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:50.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:50.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:50.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:50.855+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:50.855+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:50.855+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:51.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:51.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:51.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:51.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:52.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:51.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:52.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:51.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:52.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:51.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:52.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:52.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:52.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:52.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:52.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:52.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:52.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:52.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:52.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:54.125 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:53.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:54.125 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:53.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:54.125 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:53.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:53.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:53.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:53.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:53.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:53.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:54.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:53.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:55.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:54.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:55.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:54.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:55.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:54.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:55.105 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:54.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:55.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:54.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:55.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:54.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:55.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:54.938+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:55.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:54.938+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:55.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:54.938+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:44:55.191+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:44:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:55.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:55.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:55.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:55.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:55.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:55.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:55.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:55.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:55.809+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:56.935+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:56.935+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:56.935+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:56.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:56.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:56.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:56.770+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:56.770+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:56.770+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:57.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:57.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:57.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:57.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:57.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:58.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:57.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:58.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:57.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:58.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:57.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:58.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:57.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:58.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:58.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:58.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:58.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:58.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:58.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:44:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:58.924+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:58.924+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:44:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:58.924+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:00.152 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:59.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:00.152 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:59.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:44:59.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:59.751+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:59.751+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:44:59.751+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:59.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:59.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:00.153 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:44:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:44:59.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:00.191+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:00.702+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:00.702+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:00.703+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:00.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:00.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:00.994 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:00.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:00.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:00.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:00.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:01.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:01.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:01.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:01.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:01.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:01.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:01.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:01.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:01.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:02.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:02.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:02.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:02.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:02.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:02.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:02.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:02.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:02.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:03.802 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:03.664+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:03.802 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:03.664+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:03.802 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:03.664+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:03.967+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:03.967+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:03.967+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:03.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:03.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:03.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:04.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:04.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:04.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:04.643+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:04.643+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:04.643+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:05.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:05.192+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 55 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:04.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:04.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:04.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:05.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:05.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:05.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:05.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:05.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:05.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:06.413 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:05.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:06.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:05.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:06.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:05.987+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:06.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:06.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:06.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:06.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:07.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:07.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:07.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:06.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:06.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:06.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:07.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:07.612+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:07.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:07.612+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:07.903 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:07.612+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:08.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:08.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:08.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:07.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:07.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:07.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:08.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:08.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:08.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:08.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:08.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:08.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:09.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:09.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:09.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:09.889 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:09.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:09.889 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:09.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:09.889 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:09.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:09.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:09.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:09.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:10.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:10.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:10.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:10.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:10.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:10.192+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 86 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:10.916 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:10.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:10.916 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:10.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:10.916 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:10.657+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.017+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:11.913 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:11.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:11.913 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:11.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:11.913 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:11.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:11.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:11.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:11.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:11.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:12.946 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:12.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:12.946 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:12.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:12.946 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:12.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:13.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:12.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:12.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:12.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:13.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:13.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:13.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:13.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:13.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:13.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:14.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:13.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:13.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:13.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.016+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.016+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.016+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:14.971 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:14.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.971 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:14.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.971 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:14.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:14.972 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:14.704+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:14.972 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:14.704+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:14.972 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:14.704+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:15.301 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:15.192+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 112 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:15.301 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:15.301 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:15.301 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:14.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:15.697+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:15.697+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:15.697+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:15.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:15.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:15.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:16.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:15.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:15.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:15.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:16.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:16.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:16.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:16.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:16.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:17.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:16.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:16.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:16.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:16.960+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:17.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:17.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:17.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:17.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:17.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:17.716+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:17.945+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:17.945+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:17.945+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:18.991 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:18.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.991 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:18.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.991 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:18.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:18.991 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:18.718+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:18.992 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:18.718+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:18.992 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:18.718+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:19.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:18.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:19.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:18.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:19.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:18.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:19.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:19.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:19.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:19.669+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:19.669+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:19.669+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:20.316 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:20.193+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 141 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:20.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:20.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:20.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:20.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:20.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:20.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:20.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:20.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:20.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:20.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:20.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:20.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:20.695+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:21.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:21.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:21.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:21.866+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:21.866+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:21.866+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:21.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:21.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:21.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:22.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:22.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:22.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:22.677+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:22.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:22.678+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:22.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:22.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:22.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:23.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:23.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:23.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:23.829 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:23.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:23.830 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:23.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:23.830 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:23.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:24.080 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:23.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.080 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:23.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.080 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:23.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:24.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:24.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:24.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:24.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:24.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:24.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:25.317 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:25.193+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 168 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:25.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:25.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:25.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:25.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:25.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:25.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:25.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:25.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:25.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:25.684+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:25.684+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:25.684+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:26.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:26.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:26.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:26.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:26.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:26.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:26.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:26.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:26.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:27.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:27.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:27.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:27.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:27.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:27.693+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:27.693+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:27.693+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:27.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:27.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:27.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:28.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:28.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:28.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:28.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:28.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:28.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:28.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:28.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:28.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:28.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:29.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:29.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:29.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:29.787+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:29.787+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:29.787+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:29.674+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:29.674+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:29.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:29.674+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:30.440 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:30.194+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 197 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:30.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:30.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:30.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:30.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:30.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:30.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:30.990 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:30.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:30.990 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:30.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:30.990 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:30.711+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:30.990 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:30.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:30.991 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:30.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:30.991 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:30.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:31.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:31.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:31.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:31.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:31.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:31.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:31.738+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:31.738+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:32.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:31.738+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:32.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:32.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:32.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:32.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:32.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:32.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:32.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:32.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:33.116 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:32.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:33.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:33.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:33.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:33.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:33.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:33.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:33.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:33.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:33.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:33.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:33.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:33.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:33.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:33.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:34.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:34.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:34.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:35.168 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:34.757+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:35.168 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:34.757+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:35.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:34.757+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:35.169 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:34.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:35.169 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:34.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:35.169 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:34.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:35.453 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:35.193+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 228 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:35.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:35.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:35.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:35.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:35.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:35.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:35.797 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:35.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:35.797 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:35.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:35.797 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:35.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:36.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:35.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:36.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:35.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:36.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:35.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:36.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:36.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:36.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:37.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:36.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:37.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:36.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:37.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:36.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:37.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:36.745+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:37.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:36.745+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:37.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:36.745+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:37.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:37.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:37.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:37.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:37.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:37.728+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:37.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:37.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:37.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:38.188+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:38.188+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:38.188+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:38.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:38.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:38.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:38.687+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:38.687+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:38.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:38.687+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:39.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:39.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:39.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:39.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:39.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:39.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:39.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:39.682+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:39.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:39.682+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:39.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:39.682+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:40.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:40.194+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 256 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:40.677+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:40.677+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:40.677+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:40.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:40.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:40.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:41.642+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:41.642+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:41.642+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:41.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:41.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:41.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:42.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:42.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:42.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:42.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:42.649+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:42.649+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:42.649+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:42.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:42.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:42.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:43.799 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:43.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:43.799 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:43.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:43.799 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:43.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:44.074 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:43.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:44.074 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:43.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:44.074 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:43.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:44.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:44.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:44.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:44.603+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:44.603+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:44.603+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:44.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:44.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:44.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:45.195+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 284 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:45.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:45.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:45.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:45.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:45.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:45.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:45.623+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 161 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:45.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:45.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:45.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:45.798+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:46.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:46.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:46.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:46.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:46.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:46.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:46.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:46.799+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:46.640+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:46.640+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:46.640+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:46.640+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:47.143+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:47.143+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:47.143+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:47.143+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:47.760+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:47.760+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:47.760+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:47.761+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:47.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:47.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:47.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:47.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:47.600+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:48.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:48.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:48.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:48.181+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:48.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:48.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:48.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:48.726+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:48.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:48.574+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:48.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:48.574+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:48.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:48.574+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:48.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:48.575+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:49.422 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:49.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:49.422 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:49.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:49.422 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:49.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:49.422 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:49.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:49.701+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:49.701+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:49.701+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:49.702+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:49.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:49.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:49.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:49.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:49.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:49.560+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:50.196+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 313 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:50.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:50.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:50.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:50.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:50.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:50.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:50.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:50.663+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:50.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:50.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:50.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:50.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:50.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:50.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:50.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:50.546+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:51.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:51.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:51.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:51.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:51.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:51.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:51.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:51.226+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:51.550+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:51.550+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:51.550+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:51.551+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:51.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:51.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:51.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:51.698+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:52.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:52.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:52.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:52.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:52.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:52.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:52.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:52.202+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:52.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:52.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:52.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:52.731+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:52.576+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:52.576+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:52.576+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:52.577+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:53.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:53.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:53.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:53.186+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:53.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:53.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:53.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:53.767+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:53.589+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:53.589+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:53.589+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:53.590+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:54.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:54.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:54.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:54.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:54.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:54.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:54.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:54.235+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:54.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:54.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:54.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:54.740+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:54.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:54.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:54.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:54.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:54.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:54.594+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:45:55.196+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 340 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:45:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:55.221+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:55.221+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:55.221+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:55.222+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:55.716 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:55.587+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:55.716 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:55.587+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:55.716 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:55.587+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:55.716 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:55.588+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:55.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:55.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:55.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:55.716+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:56.633 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:56.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:56.633 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:56.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:56.633 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:56.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:56.633 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:56.211+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:56.632+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:56.632+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:56.632+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:56.633+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:56.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:56.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:56.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:56.749+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:56.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:56.750+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:57.622 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:57.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:57.622 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:57.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:57.622 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:57.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:57.622 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:57.220+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:57.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:57.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:57.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:57.731+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:57.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:57.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:57.622+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:57.622+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:58.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:58.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:58.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:58.194+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:58.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:58.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:58.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:58.734+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:58.592+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:58.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:58.592+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:58.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:58.592+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:58.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:58.593+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:59.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:59.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:59.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:45:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:45:59.196+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:59.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:59.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:59.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:45:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:45:59.694+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:45:59.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:59.629+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:59.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:59.629+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:59.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:59.629+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:45:59.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:45:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:45:59.630+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:00.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:00.197+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 369 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:00.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:00.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:00.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:00.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:00.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:00.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:00.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:00.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:00.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:00.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:00.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:00.601+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:00.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:00.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:00.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:00.721+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:01.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:01.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:01.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:01.181+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:01.597+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:01.597+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:01.597+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:01.597+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:01.713+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:01.713+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:01.713+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:01.714+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:02.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:02.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:02.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:02.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:02.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:02.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:02.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:02.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:02.623+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:02.623+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:02.623+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:02.624+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:02.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:02.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:02.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:02.698+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:03.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:03.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:03.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:03.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:03.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:03.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:03.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:03.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:03.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:03.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:03.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:03.610+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:03.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:03.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:03.715+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:03.715+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:04.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:04.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:04.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:04.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:04.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:04.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:04.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:04.208+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:04.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:04.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:04.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:04.690+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:04.561+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:04.561+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:04.561+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:04.562+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:05.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:05.197+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 397 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:05.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:05.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:05.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:05.189+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:05.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:05.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:05.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:05.689+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:05.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:05.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:05.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:05.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:05.564+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:06.181+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:06.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:06.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:06.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:06.516+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:06.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:06.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:06.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:06.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:06.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:07.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:07.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:07.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:07.192+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:07.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:07.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:07.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:07.537+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:07.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:07.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:07.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:07.637+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:08.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:08.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:08.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:08.174+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:08.587+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:08.587+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:08.587+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:08.588+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:08.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:08.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:08.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:08.569+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:09.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:09.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:09.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:09.170+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:09.953 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:09.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:09.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:09.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:09.620+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:09.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:09.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:09.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:09.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:09.608+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:10.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:10.198+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 426 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:10.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:10.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:10.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:10.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:10.585+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:10.585+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:10.585+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:10.586+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:10.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:10.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:10.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:10.607+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:11.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:11.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:11.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:11.184+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:11.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:11.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:11.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:11.559+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:11.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:11.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:11.620+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:11.621+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:12.601 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:12.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:12.601 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:12.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:12.601 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:12.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:12.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:12.234+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:12.614+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:12.614+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:12.614+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:12.614+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:12.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:12.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:12.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:12.601+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:13.577 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:13.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:13.577 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:13.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:13.577 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:13.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:13.577 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:13.283+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:13.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:13.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:13.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:13.577+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:13.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:13.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:13.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:13.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:13.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:13.600+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:13.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:13.601+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:14.572 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:14.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:14.572 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:14.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:14.572 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:14.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:14.572 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:14.259+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:14.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:14.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:14.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:14.627+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:14.571+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:14.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:14.571+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:14.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:14.571+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:14.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:14.572+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:15.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:15.198+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 453 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:15.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:15.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:15.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:15.237+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:15.582+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:15.582+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:15.582+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:15.583+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:15.581+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:15.581+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:15.581+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:15.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:15.581+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:16.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:16.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:16.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:16.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:16.261+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:16.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:16.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:16.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:16.550+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:16.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:16.579+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:16.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:16.579+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:16.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:16.579+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:16.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:16.580+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:17.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:17.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:17.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:17.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:17.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:17.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:17.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:17.294+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:17.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:17.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:17.626+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:17.627+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:17.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:17.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:17.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:17.600+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:18.607 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:18.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:18.607 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:18.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:18.607 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:18.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:18.608 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:18.315+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:18.607+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:18.607+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:18.607+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:18.607+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:18.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:18.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:18.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:18.645+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:19.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:19.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:19.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:19.587+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:19.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:19.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:19.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:19.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:19.349+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:19.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:19.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:19.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:19.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:19.695+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:20.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:20.199+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 486 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:20.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:20.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:20.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:20.309+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:20.924 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:20.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:20.924 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:20.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:20.924 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:20.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:20.924 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:20.662+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:20.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:20.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:20.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:20.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:20.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:20.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:20.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:20.542+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:21.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:21.515+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:21.515+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:21.515+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:21.516+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:21.303+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:21.303+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:21.303+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:21.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:21.303+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:21.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:21.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:21.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:21.676+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:22.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:22.486+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:22.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:22.486+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:22.486+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:22.487+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:22.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:22.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:22.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:22.692 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:22.317+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:22.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:22.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:22.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:22.692+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:23.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:23.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:23.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:23.461+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:23.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:23.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:23.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:23.293+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:23.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:23.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:23.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:23.659+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:24.641 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:24.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:24.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:24.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:24.246+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:24.458+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:24.458+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:24.458+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:24.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:24.459+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:24.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:24.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:24.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:24.642+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:25.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:25.199+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 519 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:25.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:25.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:25.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:25.251+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:25.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:25.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:25.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:25.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:25.472+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:25.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:25.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:25.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:25.668+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:26.634 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:26.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:26.634 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:26.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:26.634 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:26.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:26.634 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:26.299+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:26.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:26.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:26.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:26.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:26.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:26.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:26.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:26.438+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:26.633+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:26.633+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:26.633+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:26.634+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:27.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:27.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:27.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:27.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:27.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:27.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:27.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:27.435+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:27.675 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:27.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:27.675 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:27.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:27.675 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:27.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:27.675 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:27.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:27.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:27.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:27.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:27.674+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:28.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:28.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:28.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:28.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:28.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:28.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:28.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:28.434+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:28.646 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:28.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:28.647 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:28.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:28.647 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:28.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:28.647 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:28.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:28.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:28.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:28.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:28.646+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:29.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:29.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:29.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:29.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:29.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:29.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:29.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:29.408+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:29.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:29.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:29.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:29.686+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:30.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:30.200+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 551 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:30.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:30.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:30.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:30.397+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:30.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:30.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:30.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:30.226+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:30.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:30.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:30.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:30.682+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:31.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:31.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:31.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:31.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:31.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:31.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:31.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:31.258+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:31.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:31.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:31.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:31.441+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:31.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:31.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:31.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:31.706+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:32.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:32.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:32.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:32.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:32.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:32.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:32.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:32.401+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:32.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:32.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:32.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:32.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:32.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:32.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:32.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:32.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:32.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:32.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:32.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:32.686+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:33.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:33.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:33.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:33.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:33.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:33.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:33.523 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:33.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:33.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:33.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:33.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:33.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:33.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:33.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:33.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:33.401+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:33.656+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:33.656+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:33.656+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:33.657+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:34.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:34.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:34.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:34.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:34.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:34.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:34.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:34.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:34.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:34.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:34.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:34.698+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:35.200+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 580 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:35.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:35.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:35.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:35.335+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:35.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:35.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:35.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:35.217+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:35.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:35.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:35.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:35.687+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:36.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:36.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:36.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:36.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:36.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:36.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:36.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:36.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:37.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:36.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:36.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:36.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:36.705+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:37.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:37.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:37.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:37.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:37.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:37.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:37.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:37.212+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:37.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:37.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:37.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:37.736+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:38.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:38.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:38.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:38.204+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:38.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:38.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:38.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:38.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:38.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:38.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:38.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:38.735+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:39.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:39.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:39.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:39.247+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:39.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:39.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:39.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:39.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:39.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:39.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:39.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:39.776+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:40.200+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 608 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:40.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:40.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:40.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:40.235+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:40.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:40.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:40.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:40.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:41.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:40.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:40.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:40.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:40.729+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:41.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:41.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:41.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:41.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:41.214+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:41.214+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:41.214+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:41.214+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:41.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:41.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:41.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:41.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:42.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:42.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:42.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:42.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:42.257+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:42.810+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:43.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:43.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:43.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:43.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:43.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:43.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:43.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:43.524 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:43.253+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:43.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:43.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:43.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:43.852+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:44.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:44.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:44.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:44.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:44.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:44.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:44.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:44.281+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:44.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:44.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:44.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:44.821+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:45.200+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 636 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:45.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:45.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:45.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:45.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:45.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:45.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:45.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:45.307+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:46.165 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:45.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:46.165 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:45.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:46.165 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:45.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:46.165 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:45.851+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:46.318+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:46.318+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:46.318+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:46.318+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:46.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:46.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:46.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:46.268+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:46.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:46.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:46.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:46.887+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:47.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:47.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:47.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:47.347+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:47.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:47.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:47.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:47.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:47.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:47.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:47.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:47.855+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:48.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:48.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:48.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:48.280+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:48.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:48.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:48.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:48.387+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:49.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:48.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:48.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:48.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:48.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:49.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:49.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:49.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:49.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:49.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:49.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:49.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:49.361+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:49.776+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:50.200+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 665 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:50.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:50.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:50.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:50.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:50.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:50.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:50.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:50.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:50.332+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:51.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:50.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:50.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:50.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:50.798+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:51.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:51.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:51.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:51.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:51.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:51.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:51.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:51.332+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:51.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:51.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:51.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:51.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:52.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:52.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:52.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:52.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:52.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:52.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:52.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:52.297+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:53.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:52.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:52.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:52.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:52.846+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:53.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:53.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:53.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:53.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:53.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:53.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:53.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:53.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:53.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:53.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:53.298+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:53.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:53.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:53.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:53.874+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:54.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:54.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:54.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:54.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:54.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:54.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:54.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:54.270+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:54.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:54.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:54.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:54.831+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:46:55.202+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 692 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:46:55.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:55.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:55.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:55.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:55.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:55.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:55.241+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:55.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:55.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:55.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:55.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:55.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:55.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:55.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:55.838+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:56.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:56.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:56.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:56.272+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:56.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:56.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:56.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:56.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:56.866+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:56.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:56.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:56.867+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:57.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:57.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:57.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:57.301+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:57.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:57.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:57.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:57.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:57.887+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:57.887+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:57.887+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:57.888+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:58.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:58.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:58.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:58.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:58.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:58.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:58.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:58.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:58.307+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:58.859+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:58.859+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:58.859+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:46:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:58.859+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:59.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:59.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:59.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:46:59.248+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:59.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:59.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:59.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:46:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:46:59.280+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:59.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:59.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:59.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:46:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:46:59.856+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:00.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:00.202+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 720 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:00.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:00.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:00.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:00.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:00.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:00.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:00.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:00.282+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:00.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:00.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:00.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:00.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:00.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:01.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:00.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:00.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:00.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:00.893+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:01.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:01.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:01.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:01.220+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:01.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:01.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:01.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:01.306+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:02.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:01.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:02.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:01.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:02.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:01.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:02.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:01.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:02.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:02.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:02.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:02.172+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:02.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:02.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:02.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:02.351+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:03.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:02.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:03.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:02.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:03.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:02.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:03.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:02.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:03.150+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:03.150+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:03.150+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:03.151+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:03.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:03.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:03.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:03.364+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:04.185 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:04.185 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:04.185 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:04.185 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:03.842+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:04.185+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:04.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:04.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:04.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:04.326+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:05.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:04.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:04.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:04.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:04.861+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:05.203+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 748 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:05.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:05.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:05.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:05.341+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:05.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:05.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:05.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:05.205+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:06.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:05.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:05.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:05.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:05.837+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:06.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:06.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:06.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:06.327+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:06.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:06.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:06.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:06.249+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:06.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:06.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:06.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:06.832+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:07.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:07.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:07.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:07.311+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:07.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:07.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:07.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:07.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:08.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:07.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:08.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:07.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:08.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:07.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:08.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:07.851+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:08.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:08.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:08.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:08.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:08.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:08.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:08.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:08.318+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:09.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:08.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:09.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:08.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:09.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:08.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:09.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:08.802+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:09.132+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:09.132+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:09.132+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:09.133+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:09.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:09.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:09.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:09.349+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:10.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:09.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:10.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:09.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:10.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:09.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:10.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:09.794+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:10.355 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:10.203+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 781 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:10.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:10.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:10.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:10.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:10.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:10.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:10.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:10.096+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:10.355+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:10.355+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:10.355+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:10.355+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:11.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:11.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:11.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:11.083+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:10.843+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:10.843+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:10.843+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:11.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:10.844+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:11.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:11.393+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:11.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:11.393+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:11.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:11.393+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:11.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:11.393+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:11.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:11.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:11.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:11.891+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:12.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:12.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:12.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:12.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:12.132+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:12.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:12.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:12.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:12.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:13.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:12.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:13.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:12.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:13.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:12.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:13.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:12.872+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:13.347+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:13.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:13.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:13.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:13.130+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:14.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:13.832+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:14.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:13.832+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:14.120 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:13.832+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:14.120 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:13.833+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:14.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:14.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:14.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:14.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:14.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:14.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:14.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:14.119+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:14.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:14.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:14.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:14.389+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:15.113 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:14.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:15.113 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:14.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:15.113 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:14.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:15.113 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:14.786+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:15.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:15.204+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 814 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:15.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:15.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:15.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:15.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:15.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:15.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:15.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:15.113+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:15.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:15.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:15.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:15.426+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:16.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:15.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:16.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:15.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:16.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:15.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:16.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:15.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:16.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:16.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:16.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:16.137+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:16.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:16.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:16.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:16.382+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:17.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:16.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:17.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:16.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:17.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:16.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:17.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:16.834+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:17.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:17.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:17.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:17.173+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:17.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:17.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:17.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:17.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:17.375+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:17.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:17.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:17.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:17.787+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:18.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:18.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:18.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:18.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:18.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:18.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:18.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:18.382+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:18.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:18.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:18.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:18.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:19.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:19.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:19.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:19.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:19.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:19.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:19.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:19.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:19.401+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:19.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:19.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:19.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:19.810+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:20.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:20.205+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 842 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:20.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:20.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:20.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:20.225+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:20.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:20.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:20.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:20.420+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:20.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:20.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:20.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:20.792+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:21.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:21.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:21.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:21.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:21.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:21.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:21.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:21.423+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:21.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:21.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:21.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:21.821+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:22.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:22.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:22.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:22.380+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:22.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:22.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:22.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:22.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:22.824+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:22.824+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:22.824+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:22.825+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:23.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:23.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:23.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:23.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:23.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:23.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:23.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:23.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:23.527 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:23.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:23.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:23.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:23.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:23.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:23.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:23.413+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:23.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:23.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:23.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:23.805+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:24.653 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:24.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:24.653 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:24.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:24.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:24.455+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:24.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:24.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:24.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:24.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:24.227+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:24.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:24.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:24.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:24.777+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:24.778+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:25.205+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 870 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:25.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:25.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:25.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:25.222+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:25.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:25.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:25.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:25.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:25.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:25.420+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:25.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:25.420+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:26.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:25.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:25.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:25.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:25.816+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:26.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:26.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:26.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:26.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:26.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:26.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:26.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:26.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:26.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:26.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:26.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:26.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:26.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:26.861+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:27.499 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:27.500 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:27.500 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:27.500 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:27.205+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:27.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:27.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:27.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:27.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:27.499+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:27.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:27.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:27.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:27.816+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:28.494 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:28.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:28.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:28.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:28.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:28.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:28.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:28.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:28.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:28.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:28.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:28.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:28.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:28.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:28.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:28.495+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:29.190 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:28.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:29.190 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:28.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:29.190 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:28.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:29.190 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:28.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:29.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:29.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:29.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:29.188+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:29.819 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:29.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:29.819 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:29.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:29.819 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:29.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:29.819 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:29.465+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:29.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:29.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:29.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:29.819+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:30.457 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:30.205+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 899 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:30.457 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:30.201+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:30.457 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:30.201+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:30.457 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:30.201+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:30.458 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:30.202+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:30.788 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:30.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:30.789 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:30.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:30.789 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:30.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:30.789 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:30.457+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:30.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:30.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:30.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:30.789+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:31.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:31.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:31.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:31.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:31.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:31.224+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:31.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:31.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:31.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:31.596 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:31.413+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:31.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:31.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:31.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:31.806+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:32.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:32.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:32.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:32.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:32.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:32.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:32.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:32.368+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:32.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:32.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:32.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:32.783+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:33.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:33.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:33.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:33.327+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:33.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:33.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:33.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:33.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:33.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:33.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:33.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:33.777+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:34.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:34.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:34.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:34.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:34.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:34.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:34.298+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:34.298+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:34.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:34.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:34.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:34.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:35.206+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 926 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:35.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:35.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:35.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:35.300+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:35.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:35.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:35.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:35.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:35.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:35.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:35.805+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:35.806+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:36.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:36.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:36.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:36.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:36.375+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:36.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:36.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:36.314+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:36.644 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:36.314+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:36.942 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:36.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.942 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:36.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.942 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:36.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:36.942 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:36.830+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:37.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:37.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:37.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:37.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:37.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:37.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:37.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:37.295+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:37.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:37.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:37.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:37.800+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:38.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:38.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:38.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:38.415+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:38.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:38.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:38.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:38.254+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:38.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:38.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:38.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:38.836+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:39.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:39.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:39.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:39.375+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:39.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:39.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:39.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:39.232+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:40.187 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:39.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:40.187 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:39.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:40.187 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:39.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:40.187 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:39.808+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:40.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:40.207+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 954 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:40.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:40.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:40.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:40.187+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:40.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:40.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:40.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:40.407+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:41.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:40.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.141 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:40.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.141 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:40.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.141 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:40.848+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:41.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:41.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:41.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:41.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:41.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:41.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:41.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:41.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:41.140+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:41.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:41.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:41.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:41.807+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:42.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:42.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:42.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:42.335+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:42.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:42.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:42.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:42.170+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:43.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:42.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:42.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:42.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:42.775+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:43.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:43.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:43.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:43.348+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:43.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:43.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:43.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:43.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:43.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:43.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:43.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:43.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:43.726+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:44.141+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:44.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:44.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:44.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:44.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:44.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:44.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:44.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:44.686+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:45.207+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 982 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:45.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:45.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:45.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:45.106+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:45.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:45.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:45.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:45.350+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:46.075 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:45.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.075 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:45.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.075 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:45.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.075 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:45.720+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:46.376 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:46.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:46.376 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:46.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:46.376 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:46.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:46.376 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:46.075+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:46.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:46.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:46.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:46.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:46.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:46.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:46.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:46.376+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:46.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:46.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:46.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:46.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:47.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:47.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:47.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:47.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:47.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:47.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:47.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:47.060+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:47.696 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:47.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:47.697 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:47.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:47.697 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:47.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:47.697 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:47.363+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:47.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:47.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:47.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:47.697+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:48.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:48.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:48.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:48.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:48.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:48.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:48.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:48.052+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:48.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:48.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:48.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:48.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:48.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:48.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:48.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:48.397+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:48.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:48.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:48.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:48.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:48.664+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:49.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:49.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:49.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:49.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:49.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:49.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:49.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:49.064+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:49.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:49.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:49.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:49.448+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:49.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:49.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:49.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:49.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:50.331 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:50.208+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1027 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:50.331 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:50.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:50.331 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:50.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:50.331 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:50.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:50.331 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:50.034+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:50.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:50.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:50.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:50.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:50.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:50.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:50.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:50.451+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:50.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:50.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:50.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:50.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:50.678+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:51.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:51.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:51.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:51.036+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:51.792 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:51.632+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:51.792 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:51.632+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:51.792 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:51.632+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:51.793 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:51.633+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:51.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:51.481+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:51.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:51.481+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:51.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:51.481+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:51.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:51.482+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:52.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:52.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:52.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:52.011+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:52.011+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:52.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:52.517+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:52.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:52.517+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:52.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:52.517+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:52.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:52.518+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:52.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:52.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:52.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:52.616+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.022+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.022+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.022+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.022+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:53.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:53.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:53.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:53.584+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:53.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:53.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:53.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:53.490+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:53.991+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:54.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:54.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:54.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:54.582+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:54.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:54.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:54.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:54.442+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:55.332 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:47:55.207+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1070 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:47:55.343 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:55.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:55.343 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:55.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:55.343 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:55.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:55.343 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:55.026+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:55.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:55.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:55.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:55.564+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:55.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:55.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:55.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:55.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:55.422+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:56.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:56.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:56.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:56.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:56.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:56.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:56.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:56.046+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:56.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:56.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:56.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:56.520+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:56.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:56.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:56.440+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:56.440+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.008+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:57.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:57.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:57.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:57.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:57.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:57.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:57.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:57.475+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:58.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:58.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:58.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:58.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:57.964+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:58.949 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:58.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:58.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:58.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:58.491+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:58.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:58.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:58.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:58.502+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:58.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:58.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:58.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:47:59.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:58.948+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:59.905 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:59.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:59.905 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:59.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:59.905 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:59.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:47:59.905 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:47:59.491+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:47:59.906 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:59.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:59.906 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:59.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:59.906 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:59.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:47:59.906 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:47:59.545+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:59.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:59.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:59.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:47:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:47:59.905+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:00.489 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:00.209+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1116 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:00.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:00.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:00.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:00.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:00.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:00.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:00.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:00.546+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:00.931 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:00.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:00.931 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:00.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:00.931 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:00.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:00.931 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:00.489+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:00.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:00.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:00.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:00.929+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:01.889 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:01.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:01.889 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:01.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:01.563+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:01.564+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:01.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:01.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:01.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:01.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:01.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:01.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:01.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:01.905+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:01.906+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:48:02.250+0000 7fde922eb700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T10:48:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T10:48:02.555 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:02.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:02.555 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:02.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:02.556 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:02.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:02.556 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:02.463+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:02.937 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:02.602+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:02.937 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:02.602+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:02.938 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:02.602+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:02.938 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:02.603+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:02.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-24T10:48:03.177 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:48:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:02.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:02.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:02.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:02.937+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:03.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:03.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:03.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:03.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:03.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:03.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:03.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:03.600+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:03.925+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:03.925+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:03.925+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:03.925+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:04.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:04.537+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:04.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:04.537+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:04.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:04.537+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:04.590 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:04.538+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:04.877 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:04.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:04.877 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:04.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:04.877 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:04.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:04.877 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:04.627+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:05.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:04.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:04.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:04.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:04.891+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:05.577 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:05.209+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1159 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:05.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:05.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:05.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:05.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:05.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:05.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:05.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:05.630+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:05.871 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:05.576+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:05.872 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:05.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:05.872 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:05.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:05.872 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:05.577+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:05.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:05.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:05.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:05.871+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:06.674 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:06.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:06.674 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:06.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:06.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:06.673+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:06.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:06.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:06.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:06.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:06.619+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:06.941 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.941 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:06.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:06.825+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:07.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:07.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:07.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:07.810+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:07.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:07.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:07.619+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:07.620+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:07.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:07.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:07.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:07.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:07.671+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:08.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:08.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:08.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:08.763+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:08.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:08.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:08.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:08.577+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:08.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:08.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:08.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:08.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:08.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:08.661+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:08.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:08.663+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:09.624+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:09.624+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:09.624+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:09.625+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:09.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:09.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:09.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:09.683+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:09.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:09.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:09.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:09.792+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:10.613 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:10.210+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1204 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:10.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:10.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:10.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:10.614+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:10.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:10.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:10.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:10.666+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:10.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:10.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:10.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:10.793+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:11.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:11.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:11.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:11.834+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:11.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:11.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:11.696+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:11.697+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:11.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:11.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:11.586+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:11.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:11.587+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:12.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:12.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:12.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:12.683+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:12.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:12.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:12.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:12.553+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:12.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:12.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:12.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:12.786+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:13.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:13.636+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:13.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:13.636+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:13.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:13.636+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:13.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:13.637+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:13.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:13.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:13.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:13.557+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:13.755+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:13.755+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:13.806 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:13.755+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:13.807 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:13.755+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:14.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:14.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:14.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:14.660+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:14.590+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:14.590+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:14.590+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:14.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:14.591+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:14.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:14.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:14.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:14.788+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:15.564 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:15.211+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1248 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:15.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:15.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:15.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:15.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:15.787+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:15.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:15.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:15.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:15.663+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:15.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:15.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:15.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:15.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:15.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:15.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:15.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:15.565+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:16.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:16.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:16.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:16.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:16.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:16.639+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:16.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:16.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:16.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:16.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:16.530+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:16.819+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:16.819+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:16.819+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:16.819+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:17.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:17.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:17.606+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:17.607+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:17.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:17.842 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:17.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:17.843 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:17.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:17.843 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:17.506+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:17.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:17.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:17.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:17.841+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:18.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:18.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:18.601+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:18.602+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:18.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:18.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:18.536+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:18.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:18.537+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:18.829 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:18.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:18.829 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:18.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:18.829 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:18.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:19.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:18.828+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:19.504+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:19.504+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:19.504+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:19.505+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:19.579+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:19.579+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:19.579+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:19.816 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:19.580+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:20.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:19.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:19.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:19.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:19.816+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:20.468 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:20.212+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1294 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:20.823 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:20.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:20.823 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:20.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:20.823 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:20.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:20.823 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:20.557+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:20.823 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:20.468+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:20.824 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:20.468+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:20.824 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:20.468+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:20.824 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:20.469+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:20.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:20.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:20.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:20.823+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:21.811 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:21.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:21.811 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:21.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:21.811 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:21.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:21.811 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:21.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:21.811 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:21.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:21.812 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:21.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:21.812 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:21.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:21.812 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:21.499+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:22.088 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:21.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:22.088 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:21.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:22.088 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:21.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:22.088 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:21.811+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:22.812 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:22.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:22.812 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:22.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:22.812 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:22.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:22.813 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:22.546+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:22.813 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:22.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:22.813 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:22.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:22.813 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:22.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:22.813 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:22.473+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:22.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:22.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:22.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:22.812+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:23.814 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:23.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:23.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:23.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:23.469+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:23.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:23.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:23.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:23.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:23.585+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:23.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:23.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:23.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:23.814+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:24.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:24.567+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:24.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:24.567+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:24.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:24.567+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:24.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:24.568+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:24.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:24.514+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:24.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:24.514+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:24.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:24.514+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:24.820 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:24.514+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:24.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:24.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:24.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:24.819+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:25.477 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:25.212+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1356 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:25.827 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:25.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:25.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:25.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:25.531+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:25.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:25.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:25.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:25.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:25.477+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:26.093 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:25.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:26.093 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:25.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:26.094 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:25.827+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:26.094 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:25.827+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:26.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:26.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:26.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:26.430+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:26.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:26.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:26.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:26.502+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:27.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:26.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:27.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:26.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:27.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:26.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:27.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:26.809+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:27.483+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:27.483+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:27.483+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:27.484+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:27.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:27.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:27.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:27.411+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:27.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:27.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:27.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:27.812+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:28.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:28.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:28.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:28.521+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:28.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:28.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:28.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:28.426+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:29.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:28.831+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:29.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:28.831+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:29.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:28.831+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:29.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:28.832+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:29.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:29.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:29.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:29.468+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:29.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:29.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:29.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:29.880 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:29.543+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:29.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:29.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:29.879+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:29.880+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:30.492 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:30.213+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1453 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:30.892 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:30.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:30.892 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:30.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:30.892 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:30.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:30.892 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:30.543+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:30.893 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:30.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:30.893 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:30.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:30.893 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:30.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:30.893 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:30.492+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:31.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:30.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:30.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:30.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:30.892+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:31.889 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:31.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:31.889 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:31.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:31.889 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:31.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:31.890 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:31.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:31.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:31.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:31.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:31.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:31.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:31.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:31.890 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:31.470+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:32.187 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:31.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:32.187 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:31.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:32.187 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:31.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:32.187 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:31.889+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:32.858 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:32.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:32.858 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:32.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:32.858 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:32.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:32.858 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:32.471+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:32.859 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:32.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:32.859 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:32.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:32.859 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:32.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:32.859 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:32.479+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:32.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:32.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:32.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:32.858+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:33.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:33.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:33.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:33.496+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:33.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:33.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:33.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:33.835 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:33.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:33.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:33.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:33.834+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:33.835+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:34.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:34.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:34.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:34.541+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:34.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:34.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:34.860 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:34.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:34.861 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:34.493+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:35.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:34.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:34.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:34.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:34.860+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:35.511 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:35.213+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1546 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:35.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:35.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:35.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:35.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:35.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:35.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:35.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:35.560+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:35.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:35.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:35.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:35.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:35.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:35.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:35.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:35.512+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:35.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:35.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:35.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:35.814+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:36.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:36.545+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:36.793 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:36.545+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:36.545+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:36.546+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:36.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:36.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:36.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:36.794 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:36.513+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:36.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:36.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:36.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:36.793+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:37.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:37.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:37.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:37.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:37.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:37.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:37.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:37.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:37.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:37.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:37.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:37.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:37.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:37.564+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:37.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:37.565+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:37.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:37.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:37.815+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:37.815+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:38.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:38.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:38.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:38.541+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:38.542+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:38.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:38.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:38.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:38.815+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:39.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:39.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:39.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:39.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:39.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:39.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:39.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:39.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:39.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:39.500+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:39.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:39.500+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:39.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:39.500+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:39.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:39.501+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:40.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:39.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:39.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:39.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:39.809+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:40.498 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:40.214+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1643 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:40.776 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:40.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:40.776 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:40.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:40.776 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:40.511+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:40.776 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:40.512+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:40.777 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:40.497+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:40.777 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:40.497+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:40.777 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:40.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:40.777 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:40.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:41.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:40.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:40.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:40.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:40.776+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:41.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:41.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:41.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:41.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:41.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:41.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:41.815 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:41.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:41.815 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:41.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:41.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:41.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:41.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:41.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:41.816 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:41.503+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:41.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:41.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:41.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:41.815+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:42.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:42.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:42.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:42.499+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:42.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:42.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:42.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:42.510+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:42.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:42.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:42.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:42.789+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:43.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:43.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:43.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:43.477+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:43.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:43.788 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:43.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:43.789 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:43.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:43.789 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:43.455+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:43.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:43.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:43.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:43.788+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:44.430+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:44.430+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:44.430+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:44.431+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:44.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:44.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:44.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:44.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:44.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:44.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:44.752+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:44.752+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:45.214+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1736 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:45.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:45.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:45.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:45.457+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:45.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:45.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:45.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:45.452+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:45.710+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:45.710+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:45.710+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:45.710+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:46.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:46.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:46.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:46.462+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:46.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:46.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:46.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:46.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:46.428+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:46.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:46.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:46.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:46.687+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:47.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:47.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:47.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:47.488+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:47.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:47.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:47.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:47.472+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:47.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:47.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:47.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:47.728+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:48.448+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:48.448+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:48.448+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:48.449+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:48.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:48.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:48.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:48.522+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:48.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:48.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:48.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:48.731+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:49.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:49.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:49.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:49.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:49.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:49.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:49.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:49.572+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:50.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:49.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:49.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:49.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:49.763+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:50.546 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:50.215+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1833 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:50.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:50.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:50.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:50.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:50.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:50.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:50.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:50.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:50.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:50.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:50.787+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:50.788+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:50.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:50.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:50.545+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:50.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:51.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:51.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:51.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:51.567+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:51.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:51.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:51.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:51.391+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:51.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:51.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:51.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:51.761+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:52.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:52.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:52.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:52.553+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:52.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:52.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:52.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:52.364+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:52.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:52.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:52.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:52.785+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:53.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:53.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:53.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:53.564+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:53.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:53.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:53.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:53.412+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:53.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:53.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:53.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:53.810+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:54.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:54.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:54.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:54.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:54.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:54.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:54.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:54.408+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:54.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:54.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:54.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:54.771+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:55.542 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:48:55.216+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 1926 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:48:55.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:55.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:55.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:55.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:55.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:55.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:55.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:55.419+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:55.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:55.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:55.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:55.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:55.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:55.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:55.805 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:55.543+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:55.805+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:55.805+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:55.805+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:55.805+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:56.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:56.714 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:56.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:56.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:56.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:56.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:56.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:56.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:56.715 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:56.514+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:56.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:56.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:56.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:56.844+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:57.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:57.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:57.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:57.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:57.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:57.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:57.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:57.511+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:57.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:57.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:57.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:57.857+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:58.396+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:58.396+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:58.396+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:58.397+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:58.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:58.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:58.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:58.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:58.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:58.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:58.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:58.533+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:58.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:58.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:58.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:48:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:58.848+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:59.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:59.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:59.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:48:59.421+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:59.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:59.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:59.510+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:48:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:48:59.511+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:59.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:59.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:59.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:48:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:48:59.817+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:00.495 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:00.216+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2023 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:00.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:00.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:00.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:00.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:00.495 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:00.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:00.496 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:00.382+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:00.802 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:00.802 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:00.803 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:00.803 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:00.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:01.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:00.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:00.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:00.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:00.802+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:01.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:01.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:01.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:01.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:01.467+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:01.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:01.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:01.814+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:01.815+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:02.488+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:02.488+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:02.488+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:02.489+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:02.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:02.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:02.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:02.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:02.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:02.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:02.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:02.765+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:03.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:03.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:03.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:03.455+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:03.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:03.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:03.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:03.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:04.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:03.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:03.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:03.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:03.791+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:04.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:04.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:04.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:04.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:04.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:04.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:04.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:04.448+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:05.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:04.833+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:04.833+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:04.833+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:04.834+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:05.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:05.216+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2115 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:05.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:05.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:05.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:05.452+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:05.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:05.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:05.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:05.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:06.157 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:06.157 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:06.157 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:06.158 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:05.869+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:06.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:06.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:06.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:06.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:06.855 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:06.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:06.855 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:06.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:06.856 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:06.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:06.856 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:06.496+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:06.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:06.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:06.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:06.855+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:07.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:07.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:07.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:07.515+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:07.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:07.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:07.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:07.315+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:07.807+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:07.807+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:07.807+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:07.807+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:08.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:08.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:08.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:08.493+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:08.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:08.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:08.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:08.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:08.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:08.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:08.793+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:08.793+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:09.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:09.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:09.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:09.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:09.481+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:09.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:09.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:09.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:09.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:10.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:09.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:09.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:09.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:09.751+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:10.218+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2212 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:10.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:10.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:10.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:10.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:10.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:10.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:10.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:10.273+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:11.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:10.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:10.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:10.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:11.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:10.788+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:11.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:11.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:11.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:11.493+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:11.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:11.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:11.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:11.287+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:11.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:11.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:11.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:11.816+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:12.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:12.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:12.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:12.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:12.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:12.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:12.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:12.335+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:12.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:12.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:12.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:12.846+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:13.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:13.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:13.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:13.521+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:13.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:13.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:13.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:13.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:13.830+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:13.830+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:13.830+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:13.830+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:14.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:14.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:14.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:14.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:14.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:14.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:14.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:14.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:14.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:14.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:14.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:14.806+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:15.514 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:15.218+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2305 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:15.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:15.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:15.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:15.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:15.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:15.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:15.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:15.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:15.838 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:15.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:15.839 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:15.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:15.839 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:15.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:15.839 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:15.515+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:15.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:15.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:15.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:15.839+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:16.483+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:16.483+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:16.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:16.484+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:16.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:16.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:16.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:16.316+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:16.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:16.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:16.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:16.849+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:17.626 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:17.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:17.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:17.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:17.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:17.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:17.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:17.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:17.627 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:17.494+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:17.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:17.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:17.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:17.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:17.825+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:18.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:18.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:18.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:18.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:18.529+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:18.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:18.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:18.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:18.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:18.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:18.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:18.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:18.838+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:19.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:19.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:19.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:19.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:19.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:19.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:19.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:19.254+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:20.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:19.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:19.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:19.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:19.825+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:20.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:20.219+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2401 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:20.529 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:20.529 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:20.529 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:20.529 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:20.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:20.872 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:20.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:20.872 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:20.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:20.873 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:20.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:20.873 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:20.529+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:20.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:20.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:20.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:21.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:20.872+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:21.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:21.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:21.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:21.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:21.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:21.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:21.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:21.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:21.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:21.525+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:22.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:21.880+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:21.880+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:21.880+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:21.880+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:22.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:22.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:22.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:22.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:22.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:22.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:22.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:22.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:22.893 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:22.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:22.894 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:22.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:22.894 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:22.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:22.894 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:22.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:22.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:22.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:22.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:22.893+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:23.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:23.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:23.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:23.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:23.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:23.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:23.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:23.277+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:23.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:23.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:23.572+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:23.572+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:24.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:23.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:23.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:23.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:23.868+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:24.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:24.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:24.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:24.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:24.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:24.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:24.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:24.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:24.856 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:24.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:24.856 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:24.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:24.857 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:24.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:24.857 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:24.565+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:25.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:24.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:24.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:24.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:24.856+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:25.528 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:25.220+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2494 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:25.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:25.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:25.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:25.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:25.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:25.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:25.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:25.300+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:25.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:25.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:25.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:25.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:25.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:25.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:25.814 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:25.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:25.813+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:25.813+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:25.813+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:25.814+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:26.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:26.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:26.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:26.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:26.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:26.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:26.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:26.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:26.860+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:26.860+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:26.860+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:27.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:26.860+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:27.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:27.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:27.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:27.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:27.478+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:27.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:27.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:27.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:27.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:27.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:27.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:27.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:27.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:27.825+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:28.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:28.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:28.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:28.251+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:28.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:28.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:28.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:28.438+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:28.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:28.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:28.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:28.846+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:29.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:29.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:29.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:29.243+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:29.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:29.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:29.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:29.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:29.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:29.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:29.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:29.425+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:29.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:29.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:29.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:30.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:29.835+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:30.220+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2591 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:30.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:30.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:30.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:30.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:30.390+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:31.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:30.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:30.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:30.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:30.846+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:31.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:31.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:31.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:31.232+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:31.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:31.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:31.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:31.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:31.382+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:31.804+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:31.804+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:31.804+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:31.804+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:32.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:32.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:32.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:32.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:32.339+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:32.339+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:32.339+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:32.339+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:32.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:32.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:32.845+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:33.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:32.845+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:33.534 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:33.303+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:33.303+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:33.303+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:33.304+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:33.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:33.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:33.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:33.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:33.222+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:33.836+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:33.836+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:33.836+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:33.837+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:34.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:34.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:34.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:34.211+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:34.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:34.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:34.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:34.317+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:35.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:34.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:34.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:34.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:34.843+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:35.221+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2683 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:35.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:35.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:35.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:35.226+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:35.302+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:35.302+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:35.302+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:35.303+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:35.837+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:35.837+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:35.837+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:36.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:35.838+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:36.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:36.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:36.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:36.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:36.274+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:36.274+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:36.274+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:36.275+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:36.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:36.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:36.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:37.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:36.865+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:37.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:37.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:37.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:37.243+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:37.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:37.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:37.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:37.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:37.300+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:37.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:37.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:37.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:38.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:37.876+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:38.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:38.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:38.231+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:38.231+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:38.288+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:38.288+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:38.288+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:38.289+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:38.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:38.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:38.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:39.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:38.899+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:39.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:39.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:39.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:39.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:39.282+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:39.282+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:39.282+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:39.282+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:39.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:39.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:39.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:39.948+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:40.222+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2780 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:40.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:40.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:40.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:40.227+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:40.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:40.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:40.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:40.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:40.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:40.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:40.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:40.286+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:41.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:41.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:41.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:41.237+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:40.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:41.256 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:40.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:41.257 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:40.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:41.257 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:40.996+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:41.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:41.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:41.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:41.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:41.947+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:42.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:42.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:42.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:42.223+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:42.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:42.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:42.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:42.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:43.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:42.914+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:43.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:42.914+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:43.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:42.914+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:43.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:42.914+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:43.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:43.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:43.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:43.227+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:43.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:43.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:43.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:43.195+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:43.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:43.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:43.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:43.884+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:44.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:44.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:44.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:44.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:44.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:44.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:44.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:44.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:45.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:44.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:45.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:44.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:45.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:44.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:45.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:44.862+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:45.223+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2872 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:45.162+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:45.162+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:45.162+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:45.162+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:45.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:45.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:45.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:45.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:46.187 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:45.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:46.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:45.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:46.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:45.848+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:46.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:45.848+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:46.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:46.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:46.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:46.188+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:46.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:47.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:46.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:47.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:46.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:47.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:46.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:47.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:46.822+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:47.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:47.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:47.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:47.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:47.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:47.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:47.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:47.188+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:48.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:47.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:48.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:47.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:48.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:47.802+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:48.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:47.802+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:48.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:48.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:48.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:48.149+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:48.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:49.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:48.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:49.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:48.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:49.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:48.784+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:49.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:48.784+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:49.177+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:49.177+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:49.177+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:49.178+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:49.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:49.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:49.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:49.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:50.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:49.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:50.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:49.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:50.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:49.792+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:50.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:49.792+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:50.223+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 2968 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:50.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:50.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:50.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:50.180+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:50.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:50.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:50.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:50.365+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:50.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:50.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:50.806+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:50.806+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:51.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:51.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:51.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:51.206+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:51.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:51.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:51.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:51.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:51.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:51.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:51.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:51.776+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:52.218+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:52.218+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:52.218+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:52.218+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:52.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:52.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:52.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:52.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:53.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:52.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:52.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:52.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:52.733+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:53.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:53.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:53.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:53.176+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:53.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:53.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:53.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:53.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:53.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:53.769+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:53.769+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:53.769+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:53.769+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:54.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:54.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:54.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:54.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:54.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:54.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:54.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:54.164+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:55.173 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:54.749+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:55.173 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:54.749+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:55.173 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:54.749+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:55.174 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:54.750+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:49:55.224+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3060 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:49:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:55.173+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:55.173+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:55.173+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:55.173+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:55.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:55.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:55.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:55.367+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:55.717+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:55.717+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:55.717+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:55.718+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:56.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:56.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:56.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:56.217+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:56.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:56.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:56.407+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:56.408+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:56.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:56.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:56.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:56.764+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:57.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:57.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:57.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:57.234+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:57.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:57.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:57.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:57.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:58.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:57.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:57.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:57.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:57.779+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:58.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:58.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:58.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:58.238+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:58.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:58.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:58.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:58.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:59.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:58.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:58.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:58.750+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:49:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:58.750+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:59.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:59.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:59.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:49:59.246+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:59.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:59.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:59.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:49:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:49:59.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:59.757+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:59.757+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:59.757+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:49:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:49:59.757+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:00.225+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3156 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:00.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:00.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:00.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:00.212+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:00.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:00.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:00.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:00.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:00.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:00.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:00.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:00.347+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:01.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:00.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:00.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:00.786+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:00.786+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:01.261+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:01.261+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:01.261+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:01.262+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:01.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:01.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:01.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:01.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:01.803+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:01.803+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:01.803+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:01.803+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:02.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:02.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:02.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:02.240+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:02.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:02.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:02.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:02.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:03.079 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:02.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:03.079 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:02.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:03.079 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:02.788+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:03.079 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:02.788+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:03.334 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:03.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:03.334 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:03.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:03.334 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:03.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:03.334 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:03.257+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:03.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:03.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:03.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:03.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:03.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:03.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:03.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:03.794+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:03.795+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:04.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:04.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:04.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:04.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:04.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:04.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:04.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:04.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:04.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:04.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:04.364+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:04.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:04.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:04.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:04.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:04.769+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:05.225+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3248 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:05.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:05.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:05.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:05.187+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:05.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:05.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:05.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:05.319+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:05.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:05.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:05.812+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:06.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:05.813+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:06.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:06.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:06.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:06.221+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:06.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:06.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:06.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:06.300+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:06.825+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:06.825+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:07.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:07.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:07.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:07.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:07.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:07.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:07.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:07.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:08.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:07.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:08.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:07.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:08.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:07.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:08.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:07.809+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:08.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:08.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:08.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:08.180+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:08.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:08.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:08.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:08.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:09.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:09.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:09.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:09.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:08.768+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:09.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:09.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:09.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:09.195+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:09.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:09.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:09.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:09.258+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:10.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:09.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:09.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:09.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:09.731+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:10.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:10.226+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3344 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:10.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:10.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:10.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:10.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:10.168+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:10.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:10.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:10.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:10.697+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:11.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:11.172+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:11.172+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:11.172+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:11.173+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:11.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:11.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:11.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:11.210+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:12.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:11.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:11.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:11.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:11.709+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:12.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:12.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:12.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:12.149+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:12.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:12.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:12.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:12.217+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:12.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:12.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:12.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:12.695+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:13.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:13.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:13.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:13.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:13.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:13.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:13.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:13.177+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:13.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:13.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:13.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:13.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:13.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:13.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:13.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:13.714+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:14.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:14.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:14.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:14.211+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:14.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:14.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:14.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:14.218+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:15.194 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:14.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.194 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:14.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.194 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:14.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.194 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:14.714+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:15.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:15.226+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3436 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:15.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:15.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:15.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:15.193+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:15.260+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:15.260+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:15.260+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:15.261+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:15.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:15.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:15.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:15.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:15.669+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:16.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:16.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:16.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:16.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:16.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:16.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:16.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:16.222+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:16.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:16.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:16.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:16.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:16.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:16.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:16.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:16.217+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:16.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:16.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:16.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:16.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:17.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:17.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:17.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:17.235+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:17.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:17.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:17.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:17.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:17.200+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:17.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:17.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:17.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:17.635+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:18.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:18.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:18.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:18.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:18.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:18.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:18.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:18.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:18.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:18.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:18.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:18.631+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:19.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:19.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:19.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:19.164+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:19.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:19.886 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:19.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:19.886 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:19.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:19.886 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:19.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:19.886 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:19.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:20.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:20.228+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3532 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:20.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:20.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:20.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:20.191+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:20.203+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:20.203+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:20.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:20.204+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:20.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:20.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:20.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:20.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:20.647+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:21.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:21.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:21.146+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:21.147+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:21.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:21.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:21.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:21.205+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:21.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:21.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:21.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:21.660+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:22.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:22.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:22.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:22.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:22.103+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:22.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:22.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:22.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:22.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:22.647+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:22.648+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:23.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:23.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:23.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:23.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:23.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:23.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:23.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:23.208+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:23.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:23.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:23.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:23.620+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:24.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:24.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:24.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:24.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:24.090+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:24.575 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:24.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:24.575 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:24.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:24.575 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:24.209+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:24.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:24.210+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:24.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:24.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:24.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:24.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:24.575+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:25.353 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:25.229+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3624 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:25.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:25.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:25.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:25.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:25.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:25.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:25.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:25.057+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:25.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:25.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:25.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:25.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:25.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:25.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:25.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:25.216+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:25.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:25.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:25.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:25.606+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:26.235+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:26.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:26.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:26.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:26.030+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:26.590+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:26.590+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:26.590+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:26.590+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:27.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:27.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:27.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:27.200+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:27.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:27.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:27.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:27.047+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:27.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:27.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:27.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:27.585+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:28.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:28.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:28.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:28.042+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:28.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:28.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:28.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:28.322 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:28.219+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:28.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:28.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:28.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:28.560+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:29.251+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:29.251+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:29.251+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:29.252+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:29.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:29.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:29.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:29.039+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:29.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:29.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:29.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:29.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:29.550+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:30.354 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:30.230+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3720 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:30.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:30.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:30.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:30.297+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:30.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:30.297+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:30.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:30.297+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:30.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:30.079+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:30.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:30.079+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:30.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:30.079+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:30.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:30.080+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:30.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:30.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:30.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:30.557+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:31.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:31.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:31.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:31.296+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:31.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:31.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:31.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:31.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:31.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:31.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:31.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:31.551+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:32.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:32.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:32.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:32.347+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:32.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:32.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:32.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:32.108+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:32.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:32.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:32.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:32.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:32.577+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:33.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:33.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:33.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:33.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:33.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:33.110+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:33.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:33.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:33.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:33.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:33.322+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:33.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:33.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:33.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:33.581+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:34.362+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:34.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:34.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:34.364+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:34.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:34.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:34.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:34.132+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:34.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:34.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:34.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:34.559+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:35.231+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3812 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:35.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:35.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:35.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:35.174+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:35.410+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:35.410+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:35.410+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:35.411+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:35.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:35.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:35.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:35.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:35.564+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:36.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:36.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:36.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:36.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:36.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:36.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:36.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:36.209+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:36.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:36.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:36.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:36.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:36.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:36.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:36.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:36.441+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:36.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:36.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:36.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:36.614+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:37.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:37.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:37.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:37.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:37.606 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:37.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:37.606 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:37.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:37.606 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:37.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:37.606 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:37.409+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:37.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:37.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:37.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:37.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:37.605+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:38.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:38.422+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:38.422+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:38.422+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:38.423+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:38.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:38.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:38.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:38.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:38.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:38.654+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:38.654+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:38.654+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:38.654+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:39.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:39.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:39.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:39.384+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:39.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:39.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:39.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:39.308+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:39.701+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:39.702+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:39.702+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:39.702+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:40.690 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:40.232+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3907 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:40.690 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:40.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:40.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:40.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:40.389+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:40.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:40.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:40.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:40.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:40.314+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:40.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:40.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:40.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:40.690+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:41.436+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:41.436+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:41.436+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:41.437+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:41.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:41.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:41.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:41.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:42.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:41.720+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:41.720+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:41.720+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:41.720+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:42.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:42.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:42.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:42.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:42.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:42.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:42.681 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:42.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:42.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:42.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:42.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:42.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:42.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:42.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:42.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:42.311+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:42.680+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:42.680+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:42.680+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:42.680+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:43.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:43.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:43.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:43.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:43.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:43.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:43.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:43.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:43.514+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:43.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:43.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:43.727+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:43.727+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:44.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:44.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:44.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:44.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:44.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:44.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:44.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:44.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:45.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:44.739+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:44.739+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:44.739+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:44.739+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:45.527 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:45.232+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 3999 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:45.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:45.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:45.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:45.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:45.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:45.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:45.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:45.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:45.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:45.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:45.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:45.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:45.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:45.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:45.714+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:45.715+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:46.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:46.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:46.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:46.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:46.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:46.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:46.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:46.348+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:47.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:46.730+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:46.730+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:46.730+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:46.730+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:47.573+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:47.573+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:47.573+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:47.574+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:47.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:47.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:47.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:47.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:47.766+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:47.766+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:47.766+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:47.767+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:48.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:48.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:48.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:48.532+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:48.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:48.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:48.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:48.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:48.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:48.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:48.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:48.732+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:48.732+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:49.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:49.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:49.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:49.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:49.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:49.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:49.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:49.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:49.566+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:49.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:49.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:49.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:49.741+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:50.537 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:50.233+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4095 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:50.537 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:50.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:50.537 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:50.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:50.537 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:50.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:50.538 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:50.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:50.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:50.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:50.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:50.538+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:50.712+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:50.712+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:50.712+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:50.713+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:51.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:51.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:51.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:51.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:51.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:51.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:51.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:51.243+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:51.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:51.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:51.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:51.566+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:51.688+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:51.688+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:51.688+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:51.689+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:52.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:52.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:52.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:52.200+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:52.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:52.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:52.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:52.615+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:52.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:52.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:52.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:52.640+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:53.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:53.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:53.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:53.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:53.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:53.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:53.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:53.222+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:53.608+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:53.608+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:53.608+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:53.609+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:53.682+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:53.682+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:53.682+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:53.682+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:54.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:54.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:54.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:54.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:54.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:54.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:54.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:54.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:54.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:54.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:54.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:54.719+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:54.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:54.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:54.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:54.570+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:50:55.234+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4187 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:50:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:55.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:55.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:55.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:55.185+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:55.729+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:55.729+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:55.729+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:55.730+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:55.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:55.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:55.615+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:55.615+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:56.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:56.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:56.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:56.193+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:56.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:56.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:56.778+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:56.778+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:56.593+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:56.593+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:56.593+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:56.593+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:57.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:57.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:57.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:57.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:57.779+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:57.779+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:57.779+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:57.779+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:57.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:57.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:57.582+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:57.583+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:58.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:58.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:58.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:58.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:58.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:58.214+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:58.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:58.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:58.808 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:58.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:58.808 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:58.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:58.808 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:58.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:58.808 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:58.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:59.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:58.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:58.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:58.808+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:50:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:58.808+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:59.506 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:59.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:59.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:59.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:59.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:59.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:50:59.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:50:59.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:50:59.761 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:59.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:59.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:59.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:59.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:59.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:50:59.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:50:59.507+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:59.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:59.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:59.761+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:50:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:50:59.761+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:00.547 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:00.235+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4283 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:00.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:00.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:00.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:00.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:00.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:00.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:00.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:00.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:00.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:00.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:00.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:00.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:00.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:00.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:00.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:00.764+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:01.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:01.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:01.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:01.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:01.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:01.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:01.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:01.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:01.811+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:01.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:01.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:01.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:01.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:01.565+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:01.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:01.565+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:02.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:02.160+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:02.160+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:02.160+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:02.161+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:02.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:02.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:02.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:02.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:02.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:02.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:02.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:02.518+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:03.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:02.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:03.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:02.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:03.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:02.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:03.190 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:02.847+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:03.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:03.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:03.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:03.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:03.189+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:03.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:03.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:03.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:03.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:03.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:03.517+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:03.819 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:03.517+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:03.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:03.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:03.818+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:03.819+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:04.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:04.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:04.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:04.178+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:04.861 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:04.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:04.861 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:04.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:04.861 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:04.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:04.862 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:04.541+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:05.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:04.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:05.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:04.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:05.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:04.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:05.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:04.861+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:05.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:05.236+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4375 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:05.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:05.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:05.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:05.197+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:05.868 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:05.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:05.868 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:05.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:05.868 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:05.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:05.868 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:05.553+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:06.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:06.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:06.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:05.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:06.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:05.868+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:06.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:06.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:06.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:06.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:06.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:06.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:06.871 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:06.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:06.872 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:06.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:06.872 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:06.513+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:07.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:06.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:07.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:06.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:07.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:06.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:07.165 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:06.871+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:07.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:07.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:07.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:07.164+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:07.899 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:07.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:07.899 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:07.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:07.899 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:07.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:07.899 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:07.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:07.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:07.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:07.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:07.899+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:08.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:08.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:08.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:08.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:08.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:08.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:08.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:08.210+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:08.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:08.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:08.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:08.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:08.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:08.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:08.930 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:08.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:09.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:08.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:08.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:08.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:08.930+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:09.491 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:09.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:09.492 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:09.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:09.492 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:09.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:09.492 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:09.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:09.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:09.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:09.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:09.492+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:10.237 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:09.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:10.237 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:09.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:10.237 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:09.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:10.238 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:09.957+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:10.539 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:10.237+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4474 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:10.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:10.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:10.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:10.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:10.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:10.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:10.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:10.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:10.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:10.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:10.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:10.540+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:11.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:11.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:11.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:11.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:11.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:11.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:11.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:11.007+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:11.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:11.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:11.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:11.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:12.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:12.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:12.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:12.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:12.021+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:12.021+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:12.021+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:12.021+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:12.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:12.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:12.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:12.505+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:13.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:13.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:13.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:13.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:13.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:13.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:13.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:13.068+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:13.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:13.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:13.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:13.512+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:14.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:14.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:14.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:14.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:14.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:14.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:14.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:14.107+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:14.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:14.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:14.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:14.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:14.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:15.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:15.238+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4571 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:15.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:15.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:15.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:15.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:15.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:15.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:15.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:15.156+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:15.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:15.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:15.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:15.554+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:16.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:16.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:16.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:16.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:16.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:16.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:16.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:16.204+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:16.583+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:16.583+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:16.583+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:16.584+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:17.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:17.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:17.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:17.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:17.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:17.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:17.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:17.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:17.246+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:17.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:17.567+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:18.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:18.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:18.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:18.229+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:18.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:18.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:18.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:18.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:18.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:18.799 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:18.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:18.799 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:18.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:18.799 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:18.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:18.799 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:18.536+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:19.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:19.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:19.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:19.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:19.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:19.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:19.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:19.220+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:19.570 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:19.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:19.570 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:19.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:19.570 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:19.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:19.570 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:19.259+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:19.568+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:19.568+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:19.568+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:19.569+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:20.567 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:20.239+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4672 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:20.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:20.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:20.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:20.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:20.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:20.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:20.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:20.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:20.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:20.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:20.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:20.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:20.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:20.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:20.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:20.246+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:20.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:20.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:20.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:20.567+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:21.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:21.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:21.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:21.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:21.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:21.221+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:21.266+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:21.266+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:21.266+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:21.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:21.266+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:21.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:21.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:21.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:21.549+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:22.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:22.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:22.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:22.272+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:22.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:22.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:22.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:22.189+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:22.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:22.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:22.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:22.533+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:23.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:23.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:23.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:23.238+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:23.237+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:23.237+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:23.237+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:23.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:23.237+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:23.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:23.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:23.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:23.847 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:23.546+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:24.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:24.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:24.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:24.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:24.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:24.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:24.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:24.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:24.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:24.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:24.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:24.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:24.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:24.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:24.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:24.249+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:24.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:24.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:24.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:24.547+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:25.239+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4770 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:25.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:25.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:25.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:25.206+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:25.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:25.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:25.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:25.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:25.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:25.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:25.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:25.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:25.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:25.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:25.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:25.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:26.499 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:26.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:26.499 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:26.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:26.499 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:26.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:26.499 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:26.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:26.500 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:26.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:26.500 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:26.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:26.500 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:26.225+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:26.500 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:26.225+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:26.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:26.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:26.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:26.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:27.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:27.478+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:27.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:27.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:27.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:27.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:27.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:27.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:27.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:27.270+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:28.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:28.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:28.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:28.317+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:28.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:28.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:28.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:28.432+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:28.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:28.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:28.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:28.232+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:29.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:29.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:29.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:29.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:29.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:29.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:29.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:29.393+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:29.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:29.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:29.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:29.198+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:30.422 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:30.241+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4865 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:30.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:30.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:30.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:30.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:30.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:30.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:30.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:30.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:30.423 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:30.166+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:30.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:30.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:30.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:30.422+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:31.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:31.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:31.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:31.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:31.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:31.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:31.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:31.171+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:31.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:31.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:31.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:31.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:31.458+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:32.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:32.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:32.258+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:32.259+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:32.191+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:32.191+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:32.191+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:32.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:32.191+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:32.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:32.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:32.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:32.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:32.447+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:33.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:33.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:33.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:33.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:33.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:33.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:33.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:33.144+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:33.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:33.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:33.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:33.465+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:34.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:34.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:34.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:34.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:34.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:34.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:34.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:34.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:34.444 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:34.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:34.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:34.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:34.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:34.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:34.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:34.158+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:34.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:34.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:34.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:34.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:34.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:35.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:35.241+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 4973 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:35.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:35.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:35.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:35.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:35.149+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:35.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:35.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:35.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:35.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:36.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:36.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:36.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:36.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:36.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:36.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:36.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:36.128+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:36.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:36.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:36.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:36.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:36.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:37.150+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:37.150+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:37.150+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:37.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:37.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:37.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:37.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:37.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:37.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:37.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:37.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:37.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:37.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:38.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:38.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:38.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:38.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:38.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:38.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:38.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:38.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:38.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:38.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:38.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:38.465+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:39.306 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:39.153+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:39.306 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:39.153+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:39.306 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:39.153+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:39.306 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:39.154+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:39.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:39.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:39.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:39.425+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:39.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:39.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:39.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:39.306+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:40.365 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:40.242+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5068 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:40.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:40.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:40.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:40.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:40.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:40.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:40.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:40.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:40.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:40.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:40.107+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:40.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:40.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:40.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:40.381+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:41.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:41.109+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:41.109+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:41.109+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:41.109+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:41.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:41.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:41.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:41.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:41.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:41.369+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:41.369+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:41.369+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:41.370+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:42.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:42.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:42.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:42.380+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:42.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:42.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:42.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:42.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:42.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:42.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:42.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:42.323+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:43.401 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:43.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:43.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:43.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:43.360+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:43.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:43.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:43.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:43.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:43.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:43.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:43.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:44.398 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:44.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:44.398 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:44.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:44.398 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:44.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:44.398 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:44.378+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:44.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:44.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:44.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:44.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:44.141+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:44.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:44.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:44.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:44.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:45.385 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:45.243+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5162 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:45.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:45.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:45.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:45.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:45.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:45.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:45.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:45.131+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:45.411+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:45.411+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:45.411+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:45.411+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:45.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:45.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:45.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:45.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:46.147+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:46.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:46.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:46.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:46.374+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:46.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:46.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:46.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:46.363+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:47.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:47.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:47.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:47.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:47.372+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:47.372+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:47.372+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:47.372+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:47.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:47.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:47.189+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:47.189+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:48.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:48.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:48.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:48.376+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:48.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:48.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:48.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:48.210+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:48.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:48.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:48.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:48.403+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:49.396 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:49.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:49.396 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:49.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:49.396 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:49.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:49.396 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:49.223+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:49.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:49.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:49.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:49.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:49.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:49.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:49.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:49.396+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:50.244+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5276 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:50.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:50.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:50.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:50.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:50.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:50.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:50.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:50.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:50.227+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:50.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:50.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:50.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:50.457+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:51.486 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:51.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:51.486 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:51.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:51.486 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:51.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:51.486 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:51.389+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:51.487 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:51.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:51.487 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:51.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:51.487 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:51.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:51.487 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:51.232+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:51.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:51.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:51.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:51.486+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:52.527 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:52.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:52.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:52.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:52.247+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:52.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:52.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:52.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:52.528 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:52.365+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:52.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:52.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:52.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:52.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:53.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:53.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:53.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:53.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:53.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:53.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:53.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:53.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:53.515 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:53.251+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:53.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:53.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:53.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:53.515+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:54.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:54.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:54.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:54.271+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:54.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:54.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:54.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:54.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:54.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:54.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:54.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:54.544+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:51:55.245+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5392 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:55.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:55.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:55.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:55.326+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:55.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:55.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:55.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:55.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:55.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:55.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:55.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:55.274+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:55.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:55.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:55.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:55.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:55.587+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:56.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:56.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:56.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:56.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:56.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:56.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:56.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:56.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:56.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:56.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:56.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:56.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:56.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:56.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:56.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:56.243+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:56.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:56.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:56.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:56.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:56.562+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:57.530 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:57.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:57.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:57.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:57.320+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:57.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:57.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:57.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:57.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:57.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:57.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:57.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:57.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:57.530+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:57.531+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:58.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:58.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:58.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:58.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:58.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:58.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:58.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:58.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:58.554 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:58.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:58.554 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:58.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:58.554 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:58.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:58.554 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:58.236+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:58.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:58.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:58.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:58.554+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:59.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:59.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:59.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:51:59.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:59.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:59.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:59.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:51:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:51:59.277+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:51:59.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:59.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:59.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:59.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:51:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:51:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:51:59.549+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:00.518 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:00.246+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5486 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:00.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:00.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:00.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:00.447+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:00.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:00.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:00.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:00.519 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:00.306+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:00.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:00.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:00.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:00.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:00.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:01.290+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:01.290+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:01.290+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:01.290+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:01.470+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:01.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:01.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:01.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:01.436+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:02.262+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:02.262+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:02.262+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:02.262+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:02.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:02.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:02.491+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:02.491+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:02.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:02.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:02.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:02.476+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:03.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:03.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:03.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:03.432+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:03.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:03.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:03.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:03.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:03.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:03.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:03.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:03.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:03.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:03.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:03.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:03.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:03.280+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:04.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:04.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:04.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:04.473+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:04.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:04.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:04.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:04.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:04.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:04.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:04.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:04.288+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:05.247+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5579 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:05.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:05.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:05.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:05.452+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:05.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:05.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:05.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:05.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:05.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:05.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:05.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:05.281+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:06.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:06.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:06.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:06.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:06.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:06.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:06.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:06.458+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:06.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:06.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:06.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:06.266+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:07.601 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:07.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:07.601 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:07.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:07.601 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:07.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:07.601 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:07.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:07.312+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:07.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:07.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:07.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:07.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:07.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:08.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:08.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:08.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:08.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:08.294+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:08.294+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:08.294+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:08.295+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:08.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:08.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:08.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:08.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:08.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:08.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:08.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:08.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:09.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:09.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:09.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:09.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:09.313+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:09.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:09.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:09.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:09.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:09.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:09.567+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:10.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:10.248+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5673 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:10.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:10.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:10.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:10.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:10.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:10.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:10.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:10.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:10.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:10.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:10.335+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:10.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:10.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:10.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:10.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:10.532+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:11.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:11.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:11.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:11.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:11.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:11.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:11.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:11.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:11.373+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:11.373+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:11.373+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:11.374+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:12.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:12.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:12.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:12.517+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:12.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:12.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:12.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:12.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:12.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:12.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:12.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:12.377+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:13.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:13.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:13.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:13.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:13.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:13.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:13.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:13.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:13.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:13.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:13.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:13.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:13.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:13.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:13.542+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:14.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:14.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:14.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:14.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:14.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:14.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:14.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:14.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:14.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:14.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:14.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:14.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:14.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:14.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:14.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:14.426+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:14.550+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:14.550+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:14.550+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:14.551+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:15.249+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5767 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:15.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:15.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:15.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:15.314+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:15.549 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:15.445+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:15.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:15.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:15.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:15.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:15.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:15.446+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:15.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:15.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:15.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:15.549+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:16.433+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:16.433+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:16.433+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:16.433+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:16.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:16.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:16.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:16.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:16.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:16.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:16.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:16.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:16.364+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:17.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:17.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:17.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:17.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:17.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:17.412+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:17.544+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:17.544+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:17.544+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:17.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:17.545+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:18.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:18.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:18.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:18.526+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:18.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:18.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:18.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:18.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:18.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:18.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:18.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:18.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:19.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:19.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:19.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:19.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:19.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:19.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:19.480+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:19.480+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:19.480+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:19.559 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:19.481+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:19.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:19.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:19.558+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:19.559+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:20.542 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:20.250+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5861 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:20.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:20.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:20.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:20.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:20.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:20.498+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:20.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:20.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:20.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:20.542+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:20.543+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:21.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:21.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:21.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:21.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:21.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:21.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:21.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:21.280+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:21.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:21.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:21.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:21.535+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:21.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:21.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:21.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:21.536+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:22.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:22.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:22.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:22.496+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:22.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:22.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:22.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:22.513 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:22.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:22.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:22.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:22.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:22.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:22.513+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:23.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:23.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:23.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:23.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:23.546 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:23.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:23.547 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:23.514+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:23.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:23.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:23.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:23.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:23.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:23.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:23.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:23.272+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:23.546+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:23.546+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:24.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:24.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:24.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:24.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:24.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:24.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:24.550 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:24.506+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:24.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:24.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:24.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:24.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:24.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:24.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:24.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:24.300+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:24.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:24.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:24.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:24.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:24.551+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:25.532 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:25.251+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 5955 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:25.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:25.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:25.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:25.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:25.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:25.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:25.285+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:25.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:25.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:25.837 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:25.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:25.837 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:25.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:25.837 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:25.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:25.837 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:25.533+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:26.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:26.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:26.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:26.335+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:26.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:26.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:26.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:26.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:26.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:26.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:26.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:26.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:26.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:26.556+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:26.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:26.557+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:27.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:27.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:27.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:27.512+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:27.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:27.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:27.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:27.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:27.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:27.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:27.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:27.444+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:28.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:28.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:28.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:28.441+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:28.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:28.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:28.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:28.550+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:28.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:28.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:28.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:28.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:29.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:29.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:29.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:29.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:29.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:29.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:29.536 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:29.473+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:29.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:29.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:29.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:29.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:29.535+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:30.522 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:30.252+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6050 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:30.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:30.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:30.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:30.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:30.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:30.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:30.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:30.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:30.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:30.522+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:30.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:30.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:30.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:30.522+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:31.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:31.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:31.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:31.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:31.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:31.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:31.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:31.481+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:31.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:31.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:31.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:31.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:31.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:31.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.052 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:31.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.052 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:31.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.053 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:31.545+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:32.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:32.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:32.595 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:32.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:32.595 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:32.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:32.595 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:32.492+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:32.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:32.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:32.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:32.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:32.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:32.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:32.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:32.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:32.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:32.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:32.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:32.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:32.593+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:33.621 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:33.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:33.621 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:33.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:33.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:33.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:33.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:33.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:33.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:33.622 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:33.306+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:33.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:33.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:33.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:33.620+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:34.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:34.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:34.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:34.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:34.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:34.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:34.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:34.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:34.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:34.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:34.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:34.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:34.663+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:35.378 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:35.252+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6144 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:35.379 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:35.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:35.379 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:35.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:35.379 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:35.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:35.379 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:35.288+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:35.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:35.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:35.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:35.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:35.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:35.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:35.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:35.501+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:35.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:35.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:35.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:35.629+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:36.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:36.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:36.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:36.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:36.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:36.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:36.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:36.241+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:36.592+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:36.592+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:36.592+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:36.592+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:36.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:36.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:36.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:36.501+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:37.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:37.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:37.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:37.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:37.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:37.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:37.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:37.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:37.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:37.260+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:37.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:37.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:37.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:37.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:37.635+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:38.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:38.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:38.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:38.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:38.286+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:38.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:38.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:38.541+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:38.542+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:38.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:38.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:38.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:38.667+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:39.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:39.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:39.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:39.543+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:39.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:39.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:39.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:39.651 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:39.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:39.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:39.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:39.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:39.651+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:40.523 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:40.254+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6238 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:40.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:40.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:40.523 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:40.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:40.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:40.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:40.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:40.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:40.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:40.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:40.650+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:40.650+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:40.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:40.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:40.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:40.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:41.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:41.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:41.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:41.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:41.312+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:41.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:41.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:41.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:41.686+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:41.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:41.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:41.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:41.483+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:42.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:42.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:42.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:42.475+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:42.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:42.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:42.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:42.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:42.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:42.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:42.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:42.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:42.674+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:43.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:43.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:43.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:43.429+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:43.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:43.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:43.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:43.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:43.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:43.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:43.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:43.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:43.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:43.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:43.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:43.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:43.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:43.718+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:44.613 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:44.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:44.613 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:44.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:44.613 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:44.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:44.613 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:44.274+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:44.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:44.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:44.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:44.382+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:44.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:44.745+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:44.745+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:44.745+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:44.745+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:45.255+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6332 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:45.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:45.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:45.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:45.235+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:45.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:45.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:45.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:45.403+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:46.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:45.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:46.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:45.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:46.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:45.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:46.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:45.721+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:46.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:46.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:46.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:46.427+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:46.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:46.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:46.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:46.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:46.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:46.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:46.718+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:47.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:46.718+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:47.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:47.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:47.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:47.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:47.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:47.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:47.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:47.166+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:47.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:47.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:47.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:47.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:47.447+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:48.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:47.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:48.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:47.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:48.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:47.758+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:48.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:47.758+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:48.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:48.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:48.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:48.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:48.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:48.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:48.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:48.123+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:48.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:48.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:48.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:48.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:48.419+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:49.112 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:48.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:49.112 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:48.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:49.112 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:48.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:49.112 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:48.752+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:49.412 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:49.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:49.412 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:49.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:49.412 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:49.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:49.413 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:49.112+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:49.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:49.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:49.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:49.413+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:50.064 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:49.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:50.065 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:49.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:50.065 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:49.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:50.065 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:49.776+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:50.380 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:50.256+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6426 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:50.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:50.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:50.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:50.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:50.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:50.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:50.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:50.065+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:50.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:50.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:50.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:50.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:51.091 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:50.799+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:51.091 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:50.799+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:51.091 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:50.799+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:51.091 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:50.799+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:51.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:51.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:51.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:51.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:51.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:51.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:51.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:51.091+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:51.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:51.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:51.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:51.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:51.446+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:52.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:51.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:52.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:51.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:52.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:51.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:52.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:51.791+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:52.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:52.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:52.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:52.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:52.139+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:52.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:52.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:52.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:52.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:52.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:52.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:52.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:52.471+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:53.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:52.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:53.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:52.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:53.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:52.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:53.168 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:52.742+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:53.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:53.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:53.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:53.168+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:53.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:53.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:53.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:53.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:53.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:54.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:53.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:54.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:53.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:54.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:53.772+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:54.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:53.773+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:54.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:54.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:54.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:54.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:54.707 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:54.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:54.707 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:54.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:54.707 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:54.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:54.707 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:54.455+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:55.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:54.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:55.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:54.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:55.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:54.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:55.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:54.777+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:52:55.257+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6519 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:52:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:55.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:55.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:55.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:55.165+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:55.751 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:55.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:55.751 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:55.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:55.752 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:55.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:55.752 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:55.477+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:56.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:55.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:56.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:55.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:56.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:55.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:56.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:55.751+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:56.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:56.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:56.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:56.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:56.143+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:56.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:56.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:56.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:56.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:56.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:56.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:56.762 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:56.486+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:57.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:56.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:57.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:56.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:57.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:56.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:57.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:56.762+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:57.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:57.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:57.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:57.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:57.757 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:57.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:57.757 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:57.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:57.757 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:57.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:57.757 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:57.453+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:58.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:57.756+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:57.756+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:57.756+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:57.757+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:58.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:58.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:58.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:58.178+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:58.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:58.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:58.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:58.473+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:58.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:58.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:58.713+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:58.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:58.713+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:59.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:59.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:59.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:52:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:52:59.154+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:59.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:59.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:59.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:52:59.513+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:59.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:59.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:59.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:52:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:52:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:52:59.685+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:00.258+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6612 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:00.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:00.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:00.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:00.137+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:00.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:00.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:00.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:00.466+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:00.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:00.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:00.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:00.711+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:01.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:01.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:01.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:01.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:01.103+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:01.724 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:01.724 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:01.725 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:01.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:01.725 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:01.469+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:02.082 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.082 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.082 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.082 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:01.724+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:02.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:02.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:02.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:02.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:02.082+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:02.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:02.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:02.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:02.748+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:02.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:02.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:02.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:02.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:02.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:02.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:02.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:02.515+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:03.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:03.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:03.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:03.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:03.057+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:03.775 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:03.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:03.775 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:03.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:03.775 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:03.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:03.775 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:03.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:04.092 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:03.775+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:04.092 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:03.775+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:04.092 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:03.775+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:04.092 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:03.775+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:04.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:04.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:04.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:04.092+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:04.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:04.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:04.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:04.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:04.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:04.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:04.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:04.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:04.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:53:04.876+0000 7f95ec4f8700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T10:53:04.878 INFO:teuthology.orchestra.run.smithi026.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T10:53:05.097 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:04.796+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.097 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:04.796+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.097 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:04.796+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.098 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:04.796+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:05.383 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:05.259+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6705 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:05.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:05.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:05.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:05.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:05.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:05.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:05.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:05.098+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:05.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:05.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:05.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:05.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:05.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:05.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:05.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:05.438+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:05.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-24T10:53:05.880 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/mon.smithi026/config 2024-03-24T10:53:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:05.780+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:05.780+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:05.780+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:05.780+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:06.265 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:06.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:06.265 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:06.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:06.265 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:06.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:06.265 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:06.131+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:06.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:06.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:06.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:06.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:06.401+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:07.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:06.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:07.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:06.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:07.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:06.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:07.047 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:06.822+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:07.330 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:07.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:07.331 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:07.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:07.331 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:07.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:07.331 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:07.158+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:07.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:07.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:07.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:07.369+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:08.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:07.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:08.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:07.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:08.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:07.811+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:08.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:07.811+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:08.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:08.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:08.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:08.352+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:08.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:08.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:08.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:08.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:08.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:08.177+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:09.171 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:09.172 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:09.172 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:08.768+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:09.172 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:08.768+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:09.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:09.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:09.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:09.172+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:09.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:09.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:09.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:09.387+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:09.747+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:09.747+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:09.747+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:09.748+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:10.260+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6798 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:10.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:10.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:10.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:10.366+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:10.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:10.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:10.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:10.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:11.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:10.765+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:39.588747+0000 front 2024-03-24T10:42:39.588651+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:11.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:10.765+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:39.588619+0000 front 2024-03-24T10:42:39.588679+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:11.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:10.765+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:39.588719+0000 front 2024-03-24T10:42:39.588708+0000 (oldest deadline 2024-03-24T10:43:04.888647+0000) 2024-03-24T10:53:11.188 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:10.766+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:11.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:11.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:11.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:11.405+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:11.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:11.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:11.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:11.188+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:12.164 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:11.760+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:12.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:12.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:12.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:12.165+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:12.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:12.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:12.400+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:12.401+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:12.789+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:13.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:13.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:13.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:13.384+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:13.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:13.201+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:13.201+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:13.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:13.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:14.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:13.829+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:14.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:14.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:14.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:14.364+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:14.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:14.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:14.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:14.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:15.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:14.790+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:15.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:15.261+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6891 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:15.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:15.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:15.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:15.345+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:15.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:15.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:15.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:15.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:16.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:15.795+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:16.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:42.066203+0000 front 2024-03-24T10:42:42.066193+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:16.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:42.066261+0000 front 2024-03-24T10:42:42.066256+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:16.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:42.066233+0000 front 2024-03-24T10:42:42.066293+0000 (oldest deadline 2024-03-24T10:43:04.366305+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:16.322+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:16.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:16.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:16.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:16.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:17.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:16.844+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:17.315+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:17.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:18.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:17.828+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:18.298+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:18.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:18.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:18.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:18.288+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:18.875+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:19.304+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:19.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:19.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:19.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:19.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:19.886+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:20.278+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:20.262+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 6985 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:20.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:20.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:20.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:20.246+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:21.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:20.840+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:21.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:21.293+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:21.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 since back 2024-03-24T10:42:40.441192+0000 front 2024-03-24T10:42:40.441180+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:21.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 since back 2024-03-24T10:42:40.441128+0000 front 2024-03-24T10:42:40.441251+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:21.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 since back 2024-03-24T10:42:40.441222+0000 front 2024-03-24T10:42:40.441204+0000 (oldest deadline 2024-03-24T10:43:06.341204+0000) 2024-03-24T10:53:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:21.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:21.854+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:22.298+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:22.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:23.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:22.836+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:23.549 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:23.304+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:23.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:23.229+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:23.845+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:24.353+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:24.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:25.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:24.873+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:25.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:25.263+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7078 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:25.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:25.329+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:26.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:25.900+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:26.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:26.340+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:26.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:27.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:26.853+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:27.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:27.373+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:27.203+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:28.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:27.900+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:28.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:28.393+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:28.251+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:29.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:28.904+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:29.443+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:29.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:30.215 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:29.953+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:30.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:30.264+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7171 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:30.394+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:30.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:31.166+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:30.939+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:31.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:31.372+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:32.327 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:32.183+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:32.327 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:31.962+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:32.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:32.327+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:33.304 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:33.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:33.304 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:32.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:33.304 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:32.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:33.304 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:32.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:33.305 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:32.969+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:33.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:33.303+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:33.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:33.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:33.996+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:34.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:33.997+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:34.323+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:34.224+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:35.265+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7264 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:35.213+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:35.029+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:35.029+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:35.029+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:35.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:35.029+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:35.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:35.320+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:36.333 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:36.167+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:36.334 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:36.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:36.334 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:36.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:36.334 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:36.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:36.334 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:36.024+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:36.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:36.333+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:37.310+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:37.181+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:37.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:37.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:37.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:37.074+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:38.277+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:38.133+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:38.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:38.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:38.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:38.040+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:39.231+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:39.130+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:39.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:39.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:39.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:39.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:40.267 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:40.124+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:40.268 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:40.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:40.268 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:40.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:40.268 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:40.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:40.268 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:40.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:40.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:40.281+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:40.267+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7358 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:41.162+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:41.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:41.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:41.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:41.315+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:41.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:41.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:41.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:41.087+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:42.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:42.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:42.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:42.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:42.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:42.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:42.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:42.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:42.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:42.048+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:42.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:42.354+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:42.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:42.354+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:42.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:42.354+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:42.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:42.355+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:43.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:43.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:43.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:43.217+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:43.326+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:43.326+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:43.326+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:43.326+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:43.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:43.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:43.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:43.093+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:44.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:44.309+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:44.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:44.309+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:44.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:44.309+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:44.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:44.309+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:44.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:44.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:44.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:44.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:44.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:44.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:45.392 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:45.267+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7450 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:45.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:45.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:45.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:45.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:45.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:45.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:45.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:45.393 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:45.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:45.394 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:45.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:45.394 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:45.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:45.394 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:45.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:45.394 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:45.287+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:46.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:46.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:46.122+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:46.122+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:46.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:46.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:46.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:46.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:46.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:46.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:46.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:46.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:46.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:46.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:46.315+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:47.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:47.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:47.093+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:47.093+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:47.310+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:47.310+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:47.310+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:47.310+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:47.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:47.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:47.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:47.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:48.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:48.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:48.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:48.287+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:48.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:48.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:48.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:48.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:48.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:48.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:48.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:48.097+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:49.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:49.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:49.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:49.112+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:49.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:49.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:49.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:49.255+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:49.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:49.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:49.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:49.283+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:50.395 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:50.269+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7544 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:50.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:50.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:50.287+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:50.288+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:50.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:50.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:50.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:50.236+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:50.396 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:50.110+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:50.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:50.110+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:50.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:50.110+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:50.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:50.110+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:51.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:51.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:51.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:51.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:51.103+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:51.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:51.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:51.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:51.192+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:51.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:51.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:51.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:51.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:52.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:52.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:52.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:52.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:52.128+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:52.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:52.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:52.194+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:52.195+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:53.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:53.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:53.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:53.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:53.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:53.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:53.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:53.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:53.181+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:53.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:53.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:53.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:53.323+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:54.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:54.203+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:54.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:54.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:54.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:54.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:54.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:54.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:54.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:54.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:55.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:55.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:55.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:55.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:55.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:55.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:55.193 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:55.157+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:55.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:53:55.270+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7637 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:55.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:55.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:55.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:55.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:55.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:56.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:56.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:56.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:56.186+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:56.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:56.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:56.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:56.170+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:56.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:56.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:56.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:56.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:57.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:57.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:57.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:57.208+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:57.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:57.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:57.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:57.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:57.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:57.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:57.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:57.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:57.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:58.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:58.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:58.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:58.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:58.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:58.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:58.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:58.187+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:58.310 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:58.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:58.310 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:58.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:58.310 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:58.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:58.310 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:58.218+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:58.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:58.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:58.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:58.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:58.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:59.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:59.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:59.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:53:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:53:59.257+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:59.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:59.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:59.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:53:59.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:59.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:59.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:59.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:53:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:53:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:53:59.190+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:00.271+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7732 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:00.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:00.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:00.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:00.288+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:00.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:00.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:00.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:00.157+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:00.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:00.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:00.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:00.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:01.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:01.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:01.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:01.250+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:01.300+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:01.300+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:01.300+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:01.301+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:01.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:02.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:02.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:02.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:02.264+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:02.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:02.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:02.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:02.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:02.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:02.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:02.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:02.214+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:03.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:03.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:03.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:03.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:03.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:03.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:03.551 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:03.206+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:03.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:03.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:03.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:03.259+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:03.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:03.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:03.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:03.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:03.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:04.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:04.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:04.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:04.349+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:04.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:04.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:04.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:04.228+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:04.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:04.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:04.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:04.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:04.212+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:05.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:05.272+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7825 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:05.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:05.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:05.220+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:05.221+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:05.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:05.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:05.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:05.236+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:05.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:05.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:05.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:05.320+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:06.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:06.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:06.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:06.180+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:06.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:06.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:06.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:06.198+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:06.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:06.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:06.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:06.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:07.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:07.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:07.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:07.200+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:07.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:07.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:07.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:07.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:07.212+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:07.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:07.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:07.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:07.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:08.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:08.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:08.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:08.240+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:08.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:08.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:08.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:08.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:08.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:08.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:08.195+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:08.196+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:09.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:09.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:09.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:09.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:09.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:09.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:09.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:09.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:09.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:09.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:09.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:09.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:10.396 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:10.273+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 7921 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:10.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:10.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:10.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:10.196+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:10.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:10.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:10.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:10.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:10.132+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:10.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:10.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:10.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:10.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:10.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:10.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:10.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:10.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:11.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:11.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:11.375+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:11.376+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:11.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:11.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:11.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:11.155+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:11.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:11.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:11.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:11.228+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:12.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:12.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:12.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:12.383+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:12.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:12.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:12.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:12.144+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:12.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:12.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:12.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:12.201+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:13.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:13.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:13.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:13.239+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:13.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:13.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:13.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:13.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:13.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:13.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:13.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:13.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:14.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:14.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:14.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:14.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:14.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:14.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:14.192 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:14.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:14.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:14.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:14.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:14.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:14.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:14.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:14.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:14.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:15.399 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:15.275+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8014 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:15.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:15.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:15.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:15.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:15.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:15.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:15.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:15.189+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:15.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:15.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:15.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:15.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:15.059+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:15.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:15.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:15.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:15.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:16.417 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:16.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:16.417 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:16.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:16.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:16.189+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:16.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:16.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:16.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:16.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:16.055+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:16.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:16.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:16.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:16.418+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:17.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:17.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:17.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:17.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:17.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:17.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:17.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:17.084+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:17.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:17.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:17.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:17.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:17.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:17.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:17.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:17.233+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:17.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:17.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:17.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:17.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:17.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:18.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:18.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:18.082+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:18.082+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:18.082+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:18.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:18.083+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:18.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:18.444+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:18.444+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:18.444+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:18.445+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:19.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:19.239+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:19.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:19.239+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:19.239+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:19.239+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:19.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:19.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:19.034+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:19.443 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:19.034+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:19.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:19.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:19.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:19.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:19.442+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:20.398 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:20.276+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8110 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:20.398 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:20.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:20.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:20.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:20.290+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:20.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:20.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:20.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:20.075+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:20.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:20.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:20.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:20.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:20.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:20.467+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:21.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:21.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:21.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:21.032+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:21.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:21.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:21.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:21.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:21.266+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:21.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:21.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:21.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:21.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:21.451+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:22.299 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:22.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:22.299 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:22.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:22.299 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:22.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:22.300 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:22.006+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:22.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:22.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:22.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:22.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:22.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:22.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:22.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:22.299+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:22.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:22.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:22.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:22.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:22.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:22.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:22.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:22.482+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:23.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:23.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:23.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:23.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:23.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:23.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:23.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:23.007+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:23.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:23.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:23.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:23.492+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:23.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:23.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:23.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:23.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:23.317+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:24.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:24.025+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:24.025+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:24.025+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:24.025+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:24.290+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:24.290+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:24.290+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:24.291+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:24.483+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:24.483+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:24.483+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:24.484+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:25.335 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:25.277+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8203 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:25.335 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:25.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:25.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:25.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:25.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:25.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:25.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:25.055+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:25.645 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:25.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:25.645 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:25.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:25.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:25.335+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:25.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:25.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:25.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:25.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:25.453+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:26.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:26.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:26.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:26.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:26.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:26.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:26.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:26.050+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:26.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:26.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:26.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:26.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:26.479+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:26.353+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:26.353+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:26.353+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:26.353+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:27.387 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:27.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:27.387 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:27.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:27.387 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:27.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:27.387 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:27.076+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:27.527+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:27.527+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:27.527+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:27.528+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:27.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:27.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:27.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:27.387+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:28.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:28.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:28.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:28.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:28.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:28.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:28.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:28.050+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:28.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:28.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:28.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:28.500+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:28.338+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:29.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:29.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:29.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:29.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.035+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:29.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:29.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:29.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:29.481+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:29.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:29.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:29.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:29.338+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:30.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:30.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:30.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:30.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:29.986+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:30.629 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:30.278+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8299 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:30.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:30.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:30.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:30.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:30.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:30.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:30.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:30.331+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:30.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:30.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:30.631 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:30.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:30.631 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:30.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:30.631 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:30.519+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:31.294 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:30.979+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:31.295 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:30.979+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:31.295 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:30.979+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:31.295 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:30.979+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:31.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:31.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:31.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:31.531+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:31.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:31.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:31.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:31.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:31.295+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:31.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:31.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:31.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:32.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:31.948+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:32.562 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:32.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:32.563 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:32.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:32.563 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:32.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:32.563 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:32.301+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:32.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:32.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:32.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:32.563+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:33.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:32.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:33.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:32.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:33.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:32.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:33.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:32.983+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:33.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:33.311+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:33.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:33.311+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:33.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:33.311+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:33.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:33.311+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:33.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:33.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:33.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:33.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:33.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:33.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:33.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:33.605+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:34.284 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:34.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:34.284 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:34.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:34.285 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:34.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:34.285 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:34.008+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:34.616 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:34.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:34.616 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:34.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:34.616 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:34.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:34.616 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:34.285+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:34.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:34.616+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:34.616+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:34.616+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:34.617+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:35.289 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:35.280+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8391 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:35.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:35.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:35.290 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:35.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:35.290 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:35.033+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:35.290 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:35.033+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:35.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:35.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:35.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:35.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:35.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:35.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:35.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:35.292+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:35.627+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:35.627+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:35.627+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:35.628+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:36.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:36.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:36.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:36.280+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:36.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:36.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:36.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:36.051+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:36.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:36.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:36.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:36.615+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:37.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:37.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:37.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:37.250+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:37.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:37.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:37.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:37.085+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:37.934 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:37.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:37.934 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:37.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:37.934 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:37.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:37.934 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:37.585+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:38.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:38.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:38.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:38.130+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:38.271+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:38.271+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:38.271+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:38.272+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:38.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:38.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:38.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:38.562+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:39.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:39.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:39.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:39.248+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:39.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:39.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:39.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:39.143+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:39.569+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:39.569+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:39.569+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:39.570+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:40.404 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:40.281+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8487 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:40.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:40.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:40.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:40.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:40.102+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:40.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:40.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:40.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:40.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:40.250+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:40.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:40.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:40.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:40.593+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:40.594+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:41.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:41.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:41.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:41.099+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:41.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:41.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:41.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:41.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:41.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:41.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:41.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:41.613+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:41.613+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:42.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:42.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:42.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:42.237+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:42.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:42.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:42.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:42.074+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:42.953 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:42.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:42.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:42.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:42.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:42.599+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:42.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:42.599+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:43.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:43.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:43.189+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:43.189+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.030+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:43.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:43.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:43.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:43.577+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:43.578+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.988+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.988+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.988+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:43.988+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:44.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:44.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:44.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:44.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:44.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:44.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:44.627 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:44.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:44.625+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:44.625+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:44.625+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:44.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:44.627+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:45.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:45.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:45.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:45.208+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:44.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:44.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:44.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:45.283 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:44.982+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:45.637 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:45.282+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8580 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:45.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:45.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:45.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:45.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:45.638+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:46.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:46.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:46.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:46.211+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:45.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:45.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:45.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:45.998+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:46.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:46.665+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:46.665+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:46.665+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:46.666+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:47.249 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:46.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:47.249 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:46.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:47.249 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:46.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:47.249 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:46.981+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:47.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:47.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:47.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:47.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:47.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:47.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:47.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:47.250+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:47.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:47.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:47.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:47.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:47.660+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:48.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:48.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:48.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:48.201+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:48.027+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:48.027+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:48.027+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:48.027+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:48.667+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:48.667+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:48.667+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:48.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:48.668+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:49.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:49.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:49.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:49.212+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:49.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:49.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:49.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:49.008+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:49.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:49.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:49.630+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:49.631+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:50.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:50.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:50.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:50.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:50.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:50.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:50.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:50.168+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:50.284 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:50.285 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:50.285 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:50.285 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:50.003+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:50.604 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:50.283+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8676 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:50.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:50.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:50.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:50.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:50.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:50.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:50.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:50.605+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:51.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:51.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:51.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:51.027+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:51.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:51.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:51.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:51.147+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:51.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:51.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:51.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:51.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:51.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:51.574+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.021+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.021+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:52.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:52.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:52.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:52.164+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:52.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:52.610+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:52.610+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:52.610+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:52.610+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:52.982+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:53.158+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:53.158+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:53.158+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:53.158+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:53.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:53.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:53.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:53.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:53.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:53.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:53.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:53.584+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:53.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:53.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:53.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:53.978+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:54.554 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:54.554 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:54.555 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:54.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:54.555 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:54.203+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:54.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:54.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:54.552+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:54.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:54.553+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:55.285 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:55.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:55.285 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:55.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:55.285 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:55.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:55.285 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:55.249+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:55.286 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:55.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:55.286 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:55.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:55.286 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:55.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:55.286 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:55.001+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:55.561 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:54:55.283+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8769 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:54:55.877 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:55.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:55.877 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:55.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:55.877 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:55.560+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:55.877 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:55.561+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:56.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:56.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:56.037+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:56.037+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:56.518 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:56.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:56.518 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:56.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:56.518 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:56.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:56.518 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:56.233+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:56.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:56.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:56.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:56.518+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:57.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:57.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:57.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:57.246+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:57.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:57.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:57.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:57.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:57.081+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:57.554+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:57.554+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:57.554+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:57.554+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:58.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:58.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:58.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:58.210+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:58.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:58.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:58.032+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:58.032+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:58.538+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:58.538+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:58.538+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:58.539+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:59.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:59.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:59.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:59.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:54:59.187+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:59.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:59.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:59.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:54:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:54:59.061+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:54:59.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:59.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:59.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:59.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:59.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:59.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:54:59.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:54:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:54:59.532+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:00.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:00.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:00.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:00.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:00.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:00.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:00.286 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:00.191+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:00.287 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:00.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:00.287 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:00.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:00.287 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:00.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:00.287 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:00.043+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:00.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:00.286+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8865 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:00.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:00.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:00.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:00.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:00.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:00.530+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:00.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:00.531+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:01.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:01.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:01.083+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:01.083+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:01.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:01.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:01.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:01.153+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:01.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:01.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:01.563+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:01.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:01.563+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:02.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:02.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:02.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:02.125+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:02.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:02.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:02.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:02.064+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:02.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:02.591+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:02.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:02.591+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:02.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:02.591+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:02.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:02.592+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:03.108+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:03.108+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:03.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:03.110+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:03.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:03.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:03.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:03.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:03.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:03.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:03.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:03.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:03.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:03.635+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:03.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:03.635+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:04.120 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:04.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:04.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:04.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:04.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:04.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:04.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:04.061+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:04.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:04.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:04.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:04.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:04.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:04.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:04.639+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:04.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:04.640+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:05.410 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:05.287+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 8957 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:05.071+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:05.071+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:05.071+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:05.072+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:05.108+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:05.108+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:05.108+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:05.411 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:05.109+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:05.688+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:05.688+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:05.688+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:05.689+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:06.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:06.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:06.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:06.084+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:06.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:06.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:06.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:06.060+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:06.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:06.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:06.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:06.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:06.660+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:07.113+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:07.113+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:07.113+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:07.114+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:07.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:07.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:07.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:07.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:07.035+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:07.679+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:07.679+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:07.679+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:07.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:07.680+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:08.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:08.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:08.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:08.134+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:08.038+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:08.038+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:08.038+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:08.038+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:09.009 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:08.708+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.009 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:08.708+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.009 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:08.708+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.010 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:08.708+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.009+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:09.141+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:09.141+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:09.141+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:09.142+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:09.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:09.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:09.659+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:09.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:09.660+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:10.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:10.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:10.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:10.142+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:10.289 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:10.290 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:09.995+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:10.644 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:10.288+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9051 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:10.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:10.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:10.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:10.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:10.644+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:10.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:10.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:10.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:10.981+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:11.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:11.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:11.133+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:11.134+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:11.669+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:11.670+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:11.670+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:11.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:11.670+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:12.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:12.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:12.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:12.144+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:11.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:11.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:11.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:11.951+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:12.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:12.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:12.683+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:12.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:12.683+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:12.928+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:12.928+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:12.928+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:12.928+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:13.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:13.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:13.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:13.183+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:13.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:13.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:13.686+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:13.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:13.687+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:13.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:13.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:13.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:13.949+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:14.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:14.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:14.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:14.226+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:14.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:14.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:14.722+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:14.723+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:14.966+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:14.966+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:14.966+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:14.966+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:15.289+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9139 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:15.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:15.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:15.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:15.230+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:16.013 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:15.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:16.013 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:15.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:16.013 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:15.748+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:16.013 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:15.749+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:16.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:16.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:16.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:16.194+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:16.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:16.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:16.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:16.013+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:17.048 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:16.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:17.048 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:16.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:17.048 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:16.759+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:17.048 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:16.760+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:17.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:17.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:17.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:17.184+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:17.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:17.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:17.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:17.048+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:18.018 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:17.763+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:18.019 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:17.763+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:18.019 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:17.763+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:18.019 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:17.764+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:18.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:18.230+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:18.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:18.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:18.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:18.019+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:18.788+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:18.788+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:18.788+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:18.789+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:19.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:19.015+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:19.015+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:19.015+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:19.015+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:19.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:19.246+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:19.246+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:19.246+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:19.246+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:19.818+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:19.818+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:19.818+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:19.819+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:20.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:20.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:20.048+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:20.048+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:20.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:20.291+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9229 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:20.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:20.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:20.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:20.237+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:21.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:21.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:21.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:21.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:21.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:21.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:21.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:21.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:21.114 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:20.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:21.114 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:20.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:21.114 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:20.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:21.114 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:20.852+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:21.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:21.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:21.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:21.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:21.201+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:22.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:22.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:22.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:22.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:21.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:22.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:21.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:22.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:21.865+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:22.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:21.865+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:22.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:22.227+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:23.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:23.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:23.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:23.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:22.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:22.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:22.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:23.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:22.851+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:23.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:23.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:23.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:23.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:23.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:23.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:23.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:23.247+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:24.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:24.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:24.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:24.047+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:23.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:23.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:23.827+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:23.828+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:24.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:24.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:24.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:24.251+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:25.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:25.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:25.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:25.036+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:24.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:24.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:25.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:24.820+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:25.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:24.820+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:25.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:25.292+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9311 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:25.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:25.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:25.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:25.217+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:25.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:25.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:25.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:25.832+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:26.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:26.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:26.035+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:26.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:26.035+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:26.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:26.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:26.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:26.171+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:27.080 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:26.797+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:27.080 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:26.797+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:27.080 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:26.797+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:27.080 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:26.798+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:27.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:27.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:27.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:27.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:27.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:27.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:27.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:27.080+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:28.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:27.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:28.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:27.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:28.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:27.784+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:28.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:27.785+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:28.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:28.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:28.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:28.232+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:28.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:28.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:28.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:28.058+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:29.027 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:28.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:29.027 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:28.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:29.027 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:28.776+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:29.027 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:28.777+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:29.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:29.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:29.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:29.196+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:29.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:29.026+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:29.027+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:29.027+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:30.077 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:29.772+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:30.077 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:29.772+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:30.077 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:29.772+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:30.077 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:29.772+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:30.415 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:30.293+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9395 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:30.174+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:30.174+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:30.174+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:30.175+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:30.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:30.416 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:30.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:30.417 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:30.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:30.417 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:30.077+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:31.061 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:30.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:31.061 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:30.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:31.061 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:30.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:31.061 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:30.740+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:31.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:31.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:31.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:31.169+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:31.061+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:31.061+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:31.061+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:31.061+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:32.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:31.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:32.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:31.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:32.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:31.739+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:32.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:31.740+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:32.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:32.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:32.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:32.140+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:32.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:32.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:32.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:32.074+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:33.096 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:32.783+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:33.096 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:32.783+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:33.097 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:32.783+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:33.097 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:32.784+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:33.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:33.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:33.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:33.130+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:33.096+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:33.096+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:33.096+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:33.097+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:34.135 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:33.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:34.135 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:33.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:34.135 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:33.806+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:34.135 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:33.807+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:34.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:34.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:34.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:34.178+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:34.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:34.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:34.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:34.135+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:35.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:34.801+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:35.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:34.801+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:35.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:34.801+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:35.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:34.802+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:35.417 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:35.294+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9481 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:35.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:35.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:35.182+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:35.183+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:35.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:35.418 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:35.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:36.108 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:35.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:36.108 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:35.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:36.108 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:35.845+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:36.108 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:35.845+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:36.365 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:36.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:36.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:36.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:36.165+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:36.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:36.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:36.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:36.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:36.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:37.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:36.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:36.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:36.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:36.880+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:37.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:37.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:37.117+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:37.117+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:37.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:37.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:37.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:37.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:37.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:37.060+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:37.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:37.061+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:38.125 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:37.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:38.125 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:37.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:38.125 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:37.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:38.125 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:37.864+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:38.125 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:38.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:38.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:38.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:38.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:38.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:38.126 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:38.105+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:38.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:38.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:38.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:38.124+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:38.125+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:39.169 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:39.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:39.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:39.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:39.105+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:38.912+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:38.913+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:38.913+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:39.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:38.913+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:39.438 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:39.169+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:39.438 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:39.169+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:39.438 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:39.169+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:39.438 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:39.170+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:40.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:40.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:40.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:40.129+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:39.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:39.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:39.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:40.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:39.887+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:40.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:40.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:40.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:40.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:40.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:40.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:40.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:40.092+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:40.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:40.295+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9571 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:41.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:41.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:41.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:41.138+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:40.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:40.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:40.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:41.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:40.920+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:41.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:41.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:41.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:41.080+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:42.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:42.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:42.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:42.133+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:41.952+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:41.952+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:41.952+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:41.952+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:42.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:42.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:42.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:42.044+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:43.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:43.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:43.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:43.076+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:43.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:43.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:43.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:43.102+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:42.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:42.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:42.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:43.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:42.910+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:43.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:43.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:43.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:43.896+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:44.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:44.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:44.053+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:44.054+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:44.057+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:44.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:44.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:44.860+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:44.861+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:45.094+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:45.094+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:45.094+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:45.095+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:45.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:45.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:45.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:45.070+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:45.581 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:45.297+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9663 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:46.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:45.880+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:45.880+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:45.880+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:45.881+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:46.028+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:46.028+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:46.028+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:46.028+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:46.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:46.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:46.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:46.049+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:46.839+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:46.839+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:46.839+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:46.840+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:47.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:47.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:47.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:47.066+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:47.056+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:47.056+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:47.056+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:47.057+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:48.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:48.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:48.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:48.049+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:47.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:47.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:47.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:47.842+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:48.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:48.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:48.064+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:48.065+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:49.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:49.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:49.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:49.083+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:48.877+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:48.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:48.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:49.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:48.878+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:49.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:49.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:49.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:49.021+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:50.080+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:50.080+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:50.080+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:50.081+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:49.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:49.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:49.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:49.894+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:50.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:50.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:50.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.003+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:50.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.003+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:50.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:50.298+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9759 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:51.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:50.983+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:51.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:51.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:51.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:51.108+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:50.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:50.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:50.909+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:50.909+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:51.984 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:51.876+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.984 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:51.876+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.984 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:51.876+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:51.984 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:51.877+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:52.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:52.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:52.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:52.066+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:51.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:51.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:51.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:51.984+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:52.866+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:52.866+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:52.866+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:52.867+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:53.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:53.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:53.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:53.115+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:52.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:52.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:52.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:52.937+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:54.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:53.912+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:54.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:54.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:54.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:54.083+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:53.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:53.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:53.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:53.878+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:54.901+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:54.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:55.073 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:54.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:55.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:54.919+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:55.074 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:54.919+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:55.422 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:55:55.299+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9851 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:55:55.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:55.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:55.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:55.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:55.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:55.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:55.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:55.073+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:56.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:56.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:56.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:56.100+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:55.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:55.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:55.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:55.937+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:55.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:55.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:55.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:55.917+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:56.925+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:56.925+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:56.925+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:56.926+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:56.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:57.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:56.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:56.946+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:56.946+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:57.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:57.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:57.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:57.125+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:57.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:57.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:57.879+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:57.879+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:57.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:57.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:57.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:58.147 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:57.950+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:58.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:58.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:58.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:58.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:58.147+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:58.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:58.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:58.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:58.852+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:58.916+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:58.916+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:59.158 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:58.916+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:55:59.159 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:58.916+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:55:59.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:59.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:59.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:59.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:55:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:55:59.158+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:59.852+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:59.852+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:59.852+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:55:59.853+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:59.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:59.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:59.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:00.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:55:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:55:59.882+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:00.424 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:00.300+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 9946 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:00.424 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:00.161+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:00.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:00.161+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:00.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:00.161+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:00.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:00.162+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:01.154 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:00.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:00.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:00.875+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:00.876+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:00.920+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:00.920+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:00.920+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:01.155 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:00.920+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:01.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:01.154+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:01.154+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:01.154+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:01.155+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:01.939+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:01.939+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:01.939+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:01.939+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:01.885+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:02.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:01.885+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:02.163 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:01.885+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:02.163 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:01.885+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:02.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:02.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:02.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:02.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:02.161+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:03.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:02.910+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:03.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:02.910+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:03.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:02.910+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:03.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:02.911+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:03.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:02.894+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:03.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:02.894+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:03.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:02.894+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:03.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:02.894+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:03.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:03.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:03.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:03.148+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:03.149+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:04.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:03.940+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:04.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:03.940+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:04.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:03.940+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:04.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:03.940+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:04.134 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:03.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:04.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:03.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:04.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:03.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:04.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:03.873+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:04.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:04.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:04.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:04.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:04.133+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:05.114 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:04.852+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:05.114 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:04.852+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:05.114 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:04.852+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:05.114 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:04.852+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:05.115 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:04.899+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:05.115 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:04.899+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:05.115 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:04.899+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:05.115 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:04.899+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:05.426 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:05.301+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10039 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:05.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:05.112+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:05.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:05.112+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:05.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:05.112+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:05.427 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:05.113+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:05.933+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:05.933+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:05.933+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:05.933+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:06.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:06.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:06.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:06.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:06.069+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:05.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:05.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:05.886+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:05.886+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:06.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:06.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:06.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:06.887+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:06.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:06.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:06.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:06.885+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:07.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:07.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:07.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:07.082+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:07.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:07.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:07.932+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:07.932+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:08.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:08.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:08.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:08.042+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:07.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:07.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:07.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:07.895+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:09.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:09.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:09.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:09.026+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:08.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:09.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:08.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:08.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:08.927+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:08.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:08.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:08.882+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:08.883+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:10.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:09.922+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:10.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:09.922+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:10.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:09.922+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:10.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:09.923+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:10.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:10.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:10.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:10.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:10.020+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:09.851+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:09.851+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:09.851+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:10.160 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:09.852+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:10.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:10.303+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10134 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:10.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:10.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:10.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:10.962+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:11.052+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:11.052+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:11.052+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:11.052+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:10.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:10.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:10.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:10.887+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:12.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:12.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:12.030+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:12.030+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:11.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:11.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:11.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:12.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:11.991+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:11.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:11.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:11.872+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:11.872+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:12.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:12.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:12.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:12.960+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:13.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:13.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:13.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:13.034+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:12.906+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:12.906+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:12.906+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:12.906+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:13.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:13.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:13.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:13.980+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:14.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:14.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:14.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:14.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:14.027+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:14.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:13.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:14.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:13.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:14.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:13.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:14.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:13.913+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:14.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:14.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:14.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:15.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:14.878+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:15.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:14.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:14.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:14.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:14.985+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.021+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:15.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:15.304+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10226 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:16.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:15.996+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:15.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:15.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:15.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:16.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:15.962+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:15.918+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:15.918+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:15.918+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:16.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:15.918+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:16.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:16.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:16.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:17.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:16.919+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:16.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:16.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:16.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:16.993+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:16.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:16.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:16.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:17.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:16.915+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:17.964+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:17.964+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:17.964+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:17.965+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:17.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:17.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:17.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:17.934+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:17.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:17.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:17.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:17.992+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:18.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:18.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:18.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:18.995+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:18.928+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:18.928+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:19.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:18.928+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:19.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:18.929+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:18.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:18.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:18.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:18.983+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:19.962+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:19.962+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:19.962+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:19.963+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:20.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:20.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:20.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:20.018+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:20.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:19.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:20.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:19.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:20.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:19.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:20.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:19.894+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:20.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:20.305+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10321 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.008+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:20.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:20.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:20.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:20.937+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:20.944+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:20.944+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:20.944+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:20.944+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:21.950 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:21.921+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:21.921+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:21.921+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:21.922+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:21.988+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:21.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:21.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:21.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:21.963+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.011+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:22.887+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:22.887+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:22.887+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:23.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:22.888+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.013+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:23.987+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:23.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:23.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:23.864+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:24.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:23.866+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:23.982+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:24.960+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:24.960+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:24.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:24.961+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:24.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:24.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:24.892+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:24.893+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:24.943+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:25.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:25.307+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10414 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:25.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:25.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:25.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:25.915+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:25.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:26.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:25.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:25.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:25.983+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:25.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:25.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:25.895+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:26.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:25.895+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:26.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:26.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:26.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:26.942+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:26.871+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:26.871+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:26.871+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:27.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:26.872+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:26.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:26.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:26.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:26.941+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:27.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:27.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:27.829+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:27.829+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:27.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:27.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:28.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:27.952+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:27.952+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:27.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:27.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:27.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:27.936+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:28.895 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:28.834+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.895 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:28.834+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.895 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:28.834+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:28.896 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:28.835+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:28.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:28.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:28.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:28.896+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:29.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:28.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:29.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:28.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:29.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:28.942+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:29.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:28.942+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:30.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:29.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:30.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:29.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:30.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:29.863+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:30.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:29.864+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:29.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:29.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:29.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:29.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:29.922+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:29.922+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:29.922+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:30.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:29.923+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:30.308+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10510 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:30.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:30.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:30.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:30.914+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:30.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:30.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:30.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:30.903+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:30.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:31.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:30.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:31.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:30.950+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:31.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:30.950+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:31.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:31.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:31.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:31.960+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:31.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:31.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:31.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:31.922+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:31.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:31.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:31.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:32.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:31.936+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:33.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:33.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:33.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:33.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:33.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:33.007+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:32.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:32.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:32.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:32.911+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:32.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:32.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:32.891+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:32.891+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:33.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:33.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:33.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:33.912+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:34.025+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:34.025+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:34.025+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:34.026+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:33.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:33.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:33.941+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:34.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:33.941+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:35.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:34.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:35.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:34.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:35.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:34.947+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:35.011 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:34.948+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:35.012 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:34.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:35.012 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:34.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:35.012 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:34.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:35.012 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:34.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:35.310 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:35.011+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:35.310 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:35.011+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:35.310 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:35.011+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:35.310 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:35.011+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:35.309+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10603 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:35.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:35.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:35.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:35.957+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:36.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:36.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:36.042+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:36.042+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:35.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:35.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:35.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:35.903+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:36.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:36.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:36.951+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:37.103 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:36.952+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.003+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.003+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.003+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.004+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:37.051+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:37.051+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:37.051+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:37.104 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:37.052+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:38.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:38.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:38.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.001+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:38.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.001+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:38.100 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:38.100 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:38.100 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.984+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:38.100 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:37.984+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:38.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:38.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:38.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:38.099+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:39.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:39.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:39.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:39.105+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:38.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:38.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:38.957+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:40.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:39.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:40.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:39.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:40.311 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:39.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:39.986+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:40.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:40.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:40.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:40.039+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:40.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:40.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:40.312 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:40.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:40.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:40.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:40.311+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10699 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:41.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:41.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:41.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:41.002+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:41.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:41.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:41.018+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:41.018+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:41.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:41.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:41.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:41.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:41.086+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.033+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.002+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.002+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.002+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.002+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:42.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:42.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:42.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:42.113+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:42.971+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:42.985+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:43.114+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:43.114+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:43.114+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:43.114+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:44.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:44.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:44.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:44.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.006+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:44.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:44.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:44.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:44.149 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.007+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:44.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:44.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:44.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:44.148+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:44.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:45.000+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:45.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:45.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:45.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:45.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:45.142+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:45.314 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:45.314 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:45.314 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.986+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:45.314 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:44.986+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:45.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:45.312+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10791 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:45.980+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:45.980+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:45.980+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:45.980+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:46.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:46.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:46.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:46.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:46.012+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:46.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:46.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:46.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:46.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:46.105+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:47.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:47.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:47.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:47.014+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:47.027+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:47.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:47.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:47.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:47.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:47.102+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:48.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:48.075+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:48.075+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:48.075+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:48.076+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:48.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:48.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:48.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:48.043+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:48.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:48.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:48.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:48.083+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:49.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:49.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:49.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:49.087+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:49.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:49.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:49.111+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:49.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:49.112+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:49.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:49.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:49.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:49.105+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:50.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:50.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:50.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:50.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:50.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:50.148+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:50.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:50.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:50.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:50.057+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:50.315 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:50.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:50.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:50.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:50.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:50.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:50.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:50.089+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:50.314+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10886 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:51.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:51.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:51.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:51.132+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:51.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:51.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:51.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:51.102+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:51.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:51.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:51.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:51.074+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:52.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:52.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:52.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:52.091+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:52.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:52.171+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:52.171+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:52.171+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:52.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:52.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:52.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:52.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:53.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:53.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:53.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:53.102+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:53.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:53.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:53.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:53.197+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:53.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:53.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:53.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:53.088+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:54.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:54.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:54.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:54.148+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:54.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:54.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:54.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:54.158+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:54.112+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:54.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:54.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:54.113+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:55.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:56:55.315+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 10978 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:56:55.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:55.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:55.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:55.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:55.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:55.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:55.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:55.158+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:55.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:55.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:55.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:55.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:55.141+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:55.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:55.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:55.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:55.441 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:55.101+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:56.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:56.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:56.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:56.159+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:56.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:56.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:56.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:56.169+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:56.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:56.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:56.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:56.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:57.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:57.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:57.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:57.134+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:57.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:57.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:57.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:57.205+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:57.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:57.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:57.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:57.135+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:58.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:58.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:58.105+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:58.106+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:58.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:58.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:58.238+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:58.239+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:58.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:58.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:58.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:58.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:59.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:59.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:59.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:56:59.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:59.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:59.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:59.257+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:56:59.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:56:59.258+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:56:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:59.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:59.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:59.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:56:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:56:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:56:59.080+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:00.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:00.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:00.294+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:00.294+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:00.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:00.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:00.317 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:00.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:00.318 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:00.061+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:00.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:00.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:00.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:00.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:00.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:00.067+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:00.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:00.068+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:00.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:00.316+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11072 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:01.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:01.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:01.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:01.090+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:01.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:01.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:01.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:01.256+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:01.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:01.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:01.046+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:01.047+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:02.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:02.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:02.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:02.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:02.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:02.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:02.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:02.066+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:02.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:02.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:02.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:02.263+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:02.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:02.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:02.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:02.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:02.076+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:03.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:03.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:03.250+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:03.251+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:03.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:03.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:03.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:03.098+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:03.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:03.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:03.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:03.089+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:04.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:04.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:04.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:04.061+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:04.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:04.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:04.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:04.059+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:04.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:04.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:04.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:04.291+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:04.292+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:05.318 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:05.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:05.318 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:05.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:05.318 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:05.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:05.319 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:05.044+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:05.319 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:05.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:05.319 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:05.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:05.319 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:05.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:05.319 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:05.075+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:05.318+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11165 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:05.318+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:05.318+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:05.318+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:05.319+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:06.337 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:06.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:06.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:06.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:06.086+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:06.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:06.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:06.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:06.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:06.117+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:06.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:06.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:06.337+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:06.338+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:07.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:07.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:07.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:07.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:07.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:07.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:07.378 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:07.098+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:07.379 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:07.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:07.379 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:07.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:07.379 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:07.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:07.379 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:07.079+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:07.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:07.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:07.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:07.379+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:08.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:08.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:08.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:08.071+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:08.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:08.398 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:08.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:08.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:08.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:08.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:08.077+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:08.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:08.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:08.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:08.398+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:09.368 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:09.368 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:09.368 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:09.368 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:09.109+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:09.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:09.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:09.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:09.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:09.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:09.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:09.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:09.066+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:09.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:09.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:09.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:09.368+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:10.371 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:10.319+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11259 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:10.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:10.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:10.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:10.120+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:10.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:10.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:10.102+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:10.372 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:10.102+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:10.371+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:10.371+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:10.371+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:10.372+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:11.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:11.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:11.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:11.147+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:11.148+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:11.329+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:11.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:11.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:11.330+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:11.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:11.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:11.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:11.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:12.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:12.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:12.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:12.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:12.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:12.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:12.372 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:12.101+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:12.373 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:12.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:12.373 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:12.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:12.373 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:12.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:12.373 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:12.141+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:12.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:12.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:12.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:12.371+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:13.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:13.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:13.098+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:13.099+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:13.134+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:13.134+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:13.134+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:13.384 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:13.135+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:13.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:13.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:13.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:13.383+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:14.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:14.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:14.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:14.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:14.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:14.090+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:14.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:14.091+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:14.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:14.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:14.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:14.366+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:14.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:14.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:14.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:14.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:15.319+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11352 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:15.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:15.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:15.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:15.140+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:15.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:15.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:15.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:15.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:15.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:15.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:15.209+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:15.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:15.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:15.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:15.406+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:16.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:16.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:16.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:16.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:16.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:16.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:16.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:16.177+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:16.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:16.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:16.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:16.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:16.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:16.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:16.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:16.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:16.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:16.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:16.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:16.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:16.445+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:17.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:17.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:17.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:17.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:17.134+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:17.230+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:17.230+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:17.230+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:17.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:17.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:17.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:17.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:17.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:17.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:17.447+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:18.254 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:18.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:18.254 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:18.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:18.254 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:18.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:18.254 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:18.164+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:18.253+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:18.253+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:18.253+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:18.253+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:18.491+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:19.474 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:19.207+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:19.207+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:19.207+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:19.208+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:19.475 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:19.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:19.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:19.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:19.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:19.475+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:20.439 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:20.322+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11446 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:20.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:20.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:20.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:20.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:20.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:20.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:20.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:20.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:20.440 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:20.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:20.440 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:20.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:20.440 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:20.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:20.440 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:20.270+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:20.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:20.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:20.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:20.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:20.439+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:21.489 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:21.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:21.489 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:21.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:21.489 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:21.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:21.490 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:21.210+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:21.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:21.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:21.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:21.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:21.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:21.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:21.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:21.304+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:21.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:21.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:21.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:21.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:22.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:22.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:22.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:22.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:22.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:22.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:22.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:22.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:22.261+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:22.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:22.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:22.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:22.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:23.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:23.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:23.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:23.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:23.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:23.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:23.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:23.210+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:23.464 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:23.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:23.464 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:23.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:23.464 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:23.259+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:23.464 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:23.260+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:23.462+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:23.462+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:23.462+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:23.463+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:24.475 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:24.475 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:24.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:24.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:24.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:24.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:24.476 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:24.292+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:24.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:24.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:24.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:24.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:25.498 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:25.323+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11539 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:25.498 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:25.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:25.498 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:25.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:25.498 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:25.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:25.498 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:25.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:25.499 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:25.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:25.499 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:25.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:25.499 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:25.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:25.499 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:25.306+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:25.497+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:25.497+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:25.497+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:25.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:26.469 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:26.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:26.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:26.216+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:26.217+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:26.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:26.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:26.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:26.470 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:26.355+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:26.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:26.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:26.469+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:26.470+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:27.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:27.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:27.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:27.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:27.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:27.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:27.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:27.341+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:27.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:27.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:27.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:27.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:27.429+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:28.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:28.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:28.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:28.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:28.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:28.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:28.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:28.308+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:28.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:28.415+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:29.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:29.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:29.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:29.430+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:29.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:29.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:29.331+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:29.332+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:29.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:29.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:29.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:29.196+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:30.325+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11633 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:30.334+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:30.334+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:30.334+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:30.334+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:30.406 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:30.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:30.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:30.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:30.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:30.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:30.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:30.184+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:30.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:30.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:30.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:30.406+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:31.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:31.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:31.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:31.220+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:31.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:31.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:31.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:31.354+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:31.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:31.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:31.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:31.390+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:32.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:32.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:32.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:32.231+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:32.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:32.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:32.304+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:32.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:32.304+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:32.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:32.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:32.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:32.419+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:33.277+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:33.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:33.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:33.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:33.324+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:33.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:33.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:33.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:33.464+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:34.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:34.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:34.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:34.340+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:34.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:34.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:34.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:34.427+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:34.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:35.326+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11726 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:35.459+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:35.459+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:35.459+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:35.460+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:35.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:35.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:35.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:35.357+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:35.285+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:35.285+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:35.285+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:35.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:36.429 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:36.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:36.429 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:36.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:36.429 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:36.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:36.429 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:36.253+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:36.430 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:36.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:36.430 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:36.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:36.430 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:36.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:36.430 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:36.342+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:36.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:36.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:36.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:36.429+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:37.601 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:37.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:37.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:37.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:37.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:37.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:37.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:37.446+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:37.447+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:37.602 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:37.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:37.603 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:37.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:37.603 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:37.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:37.603 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:37.364+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:38.266+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:38.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:38.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:38.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:38.407+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:38.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:38.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:39.513 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:39.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:39.513 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:39.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:39.513 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:39.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:39.513 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:39.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:39.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:39.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:39.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:39.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:39.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:39.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:39.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:39.310+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:39.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:39.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:39.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:39.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:39.513+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:40.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:40.327+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11820 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:40.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:40.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:40.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:40.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:40.377+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:40.377+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:40.377+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:40.377+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:40.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:40.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:40.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:40.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:40.486+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:41.526 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:41.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:41.526 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:41.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:41.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:41.359+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:41.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:41.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:41.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:41.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:41.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:41.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:41.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:41.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:41.526+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:41.527+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:42.398+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:42.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:42.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:42.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:42.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:42.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:42.274+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:42.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:42.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:42.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:42.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:42.540+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:43.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:43.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:43.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:43.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:43.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:43.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:43.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:43.497+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:43.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:43.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:43.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:43.254+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:43.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:43.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:43.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:43.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:43.438+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:44.484 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:44.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:44.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:44.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:44.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:44.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:44.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:44.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:44.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:44.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:44.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:44.540+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:44.541+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:44.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:44.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:44.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:44.484+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:45.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:45.329+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 11913 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:45.181+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:45.181+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:45.181+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:45.181+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:45.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:45.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:45.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:45.474+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:45.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:45.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:45.506+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:45.507+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:46.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:46.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:46.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:46.173+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:46.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:46.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:46.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:46.481+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:46.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:46.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:46.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:46.549+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:47.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:47.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:47.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:47.184+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:47.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:47.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:47.461+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:47.461+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:47.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:47.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:47.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:47.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:48.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:48.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:48.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:48.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:48.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:48.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:48.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:48.218+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:48.548 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:48.548 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:48.548 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:48.548 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:48.439+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:48.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:48.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:48.547+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:48.548+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:49.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:49.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:49.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:49.171+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:49.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:49.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:49.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:49.544+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:49.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:49.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:49.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:49.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:49.459+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:50.330+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12007 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:50.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:50.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:50.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:50.190+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:50.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:50.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:50.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:50.494+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:50.708 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:50.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:50.708 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:50.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:50.708 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:50.543+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:50.708 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:50.544+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:51.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:51.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:51.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:51.157+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:51.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:51.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:51.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:51.525+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:51.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:51.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:51.458+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:51.459+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:52.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:52.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:52.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:52.412+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:52.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:52.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:52.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:52.178+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:52.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:52.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:52.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:52.536+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:53.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:53.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:53.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:53.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:53.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:53.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:53.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:53.141+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:53.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:53.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:53.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:53.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:53.418+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:53.560+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:53.560+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:53.560+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:53.561+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:54.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:54.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:54.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:54.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:54.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:54.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:54.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:54.107+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:54.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:54.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:54.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:54.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:54.580+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:54.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:54.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:54.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:54.428+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:55.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:57:55.331+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12100 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:57:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:55.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:55.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:55.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:55.143+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:55.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:55.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:55.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:55.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:55.457+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:55.457+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:55.457+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:55.457+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:56.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:56.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:56.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:56.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:56.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:56.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:56.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:56.154+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:56.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:56.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:56.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:56.623+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:56.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:56.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:56.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:56.427+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:57.430 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:57.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:57.430 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:57.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:57.430 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:57.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:57.430 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:57.114+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:57.678 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:57.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:57.678 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:57.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:57.678 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:57.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:57.678 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:57.427+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:57.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:57.590+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:57.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:57.590+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:57.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:57.590+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:57.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:57.591+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:58.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:58.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:58.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:58.165+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:58.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:58.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:58.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:58.399+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:58.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:58.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:58.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:58.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:58.639+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:59.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:59.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:59.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:57:59.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:59.360+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:59.360+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:59.360+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:57:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:57:59.360+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:57:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:59.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:59.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:59.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:57:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:57:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:57:59.653+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:00.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:00.333+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12194 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:00.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:00.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:00.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:00.345+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:00.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:00.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:00.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:00.167+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:00.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:00.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:00.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:00.699+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:01.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:01.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:01.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:01.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:01.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:01.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:01.682 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:01.347+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:01.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:01.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:01.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:01.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:01.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:01.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:01.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:01.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:01.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:01.682+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:02.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:02.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:02.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:02.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:02.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:02.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:02.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:02.220+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:02.628 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:02.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:02.628 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:02.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:02.628 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:02.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:02.628 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:02.397+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:02.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:02.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:02.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:02.683+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:03.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:03.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:03.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:03.183+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:03.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:03.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:03.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:03.359+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:03.821 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:03.653+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:03.821 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:03.653+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:03.821 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:03.653+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:03.821 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:03.653+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:04.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:04.185+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:04.319+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:04.319+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:04.319+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:04.320+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:04.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:04.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:04.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:04.631+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:05.663 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:05.334+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12287 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:05.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:05.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:05.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:05.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:05.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:05.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:05.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:05.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:05.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:05.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:05.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:05.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:05.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:05.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:05.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:05.282+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:05.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:05.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:05.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:05.664+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:06.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:06.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:06.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:06.218+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:06.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:06.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:06.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:06.663 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:06.240+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:06.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:06.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:06.662+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:06.663+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:07.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-24T10:58:07.556+0000 7f2ef0ac8700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T10:58:07.557 INFO:teuthology.orchestra.run.smithi026.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:07.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:07.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:07.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:07.271+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:07.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:07.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:07.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:07.558 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:07.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:07.901 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:07.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:07.902 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:07.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:07.902 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:07.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:07.902 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:07.678+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:58:08.381 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T10:58:08.592 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c83b54c654284040abb8b56eb8888318 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T10:58:08.596 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T10:58:08.608 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T10:58:08.608 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:58:08.608 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:58:08.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:08.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:08.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:08.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:08.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:08.282+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:08.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:08.283+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:08.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:08.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:08.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:08.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:08.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:08.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:08.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:08.157+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:08.631 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T10:58:08.631 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:58:08.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:58:09.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:08.714+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:09.110 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:08.714+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:09.110 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:08.714+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:09.110 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:08.715+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:09.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:09.110+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:09.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:09.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:09.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:09.283+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:10.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:09.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:09.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:09.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:09.709+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:10.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:10.336+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12381 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:10.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:10.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:10.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:10.287+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:10.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:10.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:10.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:10.123+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:10.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:10.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:10.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:10.674+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:11.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:11.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:11.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:11.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:11.138+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:11.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:11.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:11.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:11.291+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:11.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:11.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:11.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:11.644+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:11.645+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:12.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:12.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:12.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:12.167+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:12.168+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:12.241+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:12.241+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:12.241+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:12.242+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:12.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:12.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:12.694+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:12.694+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:13.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:13.194+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:13.194+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:13.194+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:13.195+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:13.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:13.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:13.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:13.219+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:13.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:13.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:13.664+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:13.665+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:14.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:14.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:14.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:14.233+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:14.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:14.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:14.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:14.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:14.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:14.648+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:14.648+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:14.648+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:14.649+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:15.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:15.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:15.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:15.338 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:15.281+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:15.339 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:15.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:15.339 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:15.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:15.339 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:15.295+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:15.339 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:15.296+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:15.670 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:15.337+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12474 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:15.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:15.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:15.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:15.670+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:16.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:16.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:16.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:16.283+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:16.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:16.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:16.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:16.299+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:16.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:16.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:16.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:16.667+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:16.667+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:17.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:17.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:17.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:17.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:17.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:17.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:17.686 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:17.291+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:17.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:17.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:17.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:17.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:17.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:17.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:17.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:17.275+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:17.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:17.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:17.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:17.686+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:18.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:18.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:18.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:18.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:18.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:18.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:18.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:18.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:18.693 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:18.257+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:18.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:18.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:18.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:18.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:18.693+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:19.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:19.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:19.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:19.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:19.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:19.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:19.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:19.291+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:19.292+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:20.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:19.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:19.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:19.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:19.724+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:20.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:20.339+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12568 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:20.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:20.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:20.257+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:20.257+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:20.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:20.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:20.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:20.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:20.687+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:20.687+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:20.687+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:20.688+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:21.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:21.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:21.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:21.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:21.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:21.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:21.684 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:21.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:21.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:21.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:21.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:21.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:21.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:21.265+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:21.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:21.266+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:21.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:21.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:21.682+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:21.683+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:22.689 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:22.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:22.689 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:22.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:22.689 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:22.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:22.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:22.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:22.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:22.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:22.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:22.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:22.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:22.288+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:22.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:22.288+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:22.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:22.688+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:22.688+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:22.688+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:22.688+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:23.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:23.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:23.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:23.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:23.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:23.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:23.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:23.294+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:23.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:23.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:23.689+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:23.690+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:24.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:24.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:24.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:24.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:24.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:24.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:24.542 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:24.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:24.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:24.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:24.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:24.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:24.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:24.251+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:24.543 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:24.252+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:24.646+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:24.646+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:24.646+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:24.647+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:25.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:25.339+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12661 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:25.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:25.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:25.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:25.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:25.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:25.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:25.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:25.258+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:25.695+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:25.695+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:25.695+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:25.695+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:26.684 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:26.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:26.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:26.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:26.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:26.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:26.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:26.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:26.685 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:26.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:26.685 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:26.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:26.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:26.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:26.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:26.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:26.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:26.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:26.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:26.684+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:27.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:27.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:27.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:27.249+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:27.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:27.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:27.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:27.683 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:27.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:27.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:27.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:27.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:27.682+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:28.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:28.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:28.686 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:28.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:28.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:28.221+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:28.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:28.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:28.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:28.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:28.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:28.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:28.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:28.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:28.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:28.687+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:29.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:29.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:29.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:29.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:29.275+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:29.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:29.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:29.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:29.245+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:30.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:29.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:29.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:29.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:29.724+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:30.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:30.341+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12755 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:30.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:30.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:30.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:30.273+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:30.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:30.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:30.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:30.280+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:30.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:30.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:30.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:30.686+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:30.687+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:31.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:31.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:31.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:31.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:31.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:31.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:31.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:31.275+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:31.638 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:31.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:31.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:31.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:31.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:31.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:31.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:31.270+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:31.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:31.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:31.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:31.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:31.638+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:32.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:32.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:32.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:32.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:32.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:32.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:32.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:32.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:32.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:32.238+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:32.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:32.238+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:32.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:32.238+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:32.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:32.239+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:32.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:32.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:32.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:32.677+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:33.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:33.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:33.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:33.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:33.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:33.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:33.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:33.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:33.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:33.243+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:33.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:33.243+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:33.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:33.243+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:33.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:33.243+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:33.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:33.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:33.670+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:33.670+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:34.541 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:34.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:34.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:34.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:34.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:34.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:34.542 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:34.250+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:34.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:34.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:34.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:34.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:34.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:34.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:34.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:34.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:34.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:34.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:34.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:34.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:34.637+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:35.652 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:35.343+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12848 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:35.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:35.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:35.245+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:35.246+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:35.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:35.653 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:35.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:35.654 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:35.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:35.654 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:35.282+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:35.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:35.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:35.652+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:35.653+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:36.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:36.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:36.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:36.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:36.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:36.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:36.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:36.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:36.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:36.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:36.232+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:36.233+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:36.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:36.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:36.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:36.699+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:36.700+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:37.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:37.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:37.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:37.614 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:37.268+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:37.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:37.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:37.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:37.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:37.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:37.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:37.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:37.323+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:37.915 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:37.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:37.916 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:37.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:37.916 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:37.683+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:37.916 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:37.683+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:38.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:38.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:38.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:38.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:38.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:38.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:38.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:38.343+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:38.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:38.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:38.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:38.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:38.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:38.224+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:38.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:38.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:38.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:38.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:38.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:38.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:38.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:38.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:38.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:38.640+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:39.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:39.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:39.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:39.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:39.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:39.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:39.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:39.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:39.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:39.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:39.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:39.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:39.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:39.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:39.594 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:39.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:39.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:39.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:39.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:39.593+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:40.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:40.344+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 12942 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:40.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:40.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:40.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:40.273+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:40.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:40.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:40.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:40.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:40.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:40.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:40.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:40.618+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:41.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:41.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:41.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:41.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:41.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:41.179+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:41.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:41.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:41.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:41.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:41.618+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:42.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:42.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:42.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:42.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:42.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:42.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:42.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:42.167+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:42.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:42.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:42.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:42.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:42.592+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:42.592+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:43.251 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:43.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:43.252 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:43.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:43.252 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:43.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:43.252 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:43.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:43.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:43.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:43.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:43.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:43.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:43.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:43.571 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:43.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:43.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:43.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:43.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:43.576+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:43.577+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:44.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:44.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:44.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:44.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:44.235+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:44.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:44.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:44.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:44.193+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:44.911 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:44.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:44.911 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:44.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:44.911 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:44.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:44.912 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:44.559+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:45.346+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13035 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:45.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:45.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:45.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:45.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:45.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:45.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:45.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:45.186+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:45.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:45.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:45.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:45.520+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:45.521+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:46.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:46.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:46.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:46.143+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:46.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:46.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:46.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:46.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:46.480 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:46.129+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:46.830 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:46.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:46.830 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:46.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:46.830 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:46.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:46.830 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:46.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:47.188+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:47.188+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:47.188+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:47.189+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:47.172+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:47.172+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:47.172+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:47.172+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:47.509+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:47.509+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:47.509+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:47.510+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:48.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:48.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:48.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:48.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:48.209+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:48.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:48.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:48.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:48.199+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:48.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:48.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:48.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:48.552+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:48.553+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:49.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:49.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:49.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:49.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:49.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:49.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:49.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:49.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:49.592 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:49.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:49.592 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:49.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:49.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:49.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:49.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:49.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:49.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:49.591+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:49.591+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:49.591+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:49.592+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:50.347+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13129 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:50.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:50.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:50.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:50.196+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:50.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:50.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:50.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:50.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:50.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:50.554+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:50.554+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:50.554+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:50.554+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:51.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:51.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:51.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:51.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:51.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:51.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:51.533 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:51.233+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:51.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:51.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:51.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:51.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:51.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:51.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:51.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:51.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:51.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:51.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:51.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:51.532+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:52.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:52.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:52.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:52.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:52.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:52.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:52.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:52.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:52.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:52.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:52.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:52.503 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:52.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:52.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:52.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:52.501+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:52.502+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:53.410 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:53.161+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:53.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:53.161+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:53.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:53.161+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:53.411 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:53.161+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:53.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:53.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:53.523+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:53.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:53.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:53.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:53.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:53.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:53.411+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:54.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:54.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:54.392 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:54.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:54.392 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:54.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:54.392 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:54.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:54.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:54.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:54.535+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:54.535+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:54.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:54.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:54.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:54.391+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:55.368 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:58:55.349+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13222 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:55.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:55.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:55.342+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:55.343+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:55.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:55.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:55.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:55.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:55.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:55.107+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:55.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:55.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:55.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:55.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:55.516+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:56.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:56.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:56.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:56.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:56.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:56.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:56.316 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:56.102+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:56.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:56.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:56.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:56.500+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:56.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:56.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:56.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:56.316+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:57.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:57.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:57.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:57.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:57.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:57.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:57.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:57.075+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:57.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:57.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:57.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:57.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:57.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:57.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:57.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:57.338+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:58.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:58.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:58.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:58.123+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:58.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:58.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:58.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:58.327+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:58.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:58.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:58.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:58.474+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:59.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:59.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:59.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:58:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:58:59.300+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:59.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:59.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:59.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:58:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:58:59.077+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:58:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:59.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:59.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:59.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:58:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:58:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:58:59.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:00.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:00.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:00.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:00.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:00.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:00.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:00.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:00.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:00.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:00.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:00.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:00.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:00.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:00.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:00.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:00.093+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:00.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:00.350+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13316 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:00.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:00.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:00.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:00.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:01.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:01.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:01.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:01.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:01.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:01.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:01.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:01.052+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:01.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:01.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:01.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:01.486+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:02.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:02.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:02.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:02.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:02.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:02.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:02.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:02.043+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:02.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:02.044+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:02.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:02.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:02.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:02.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:02.452+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:03.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:03.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:03.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:03.221+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:03.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:03.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:03.075+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:03.076+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:03.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:03.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:03.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:03.484+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:04.200 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:04.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:04.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:04.084+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:04.084+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:04.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:04.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:04.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:04.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:04.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:04.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:04.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:04.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:04.493+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:05.352 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:05.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:05.352 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:05.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:05.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:05.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:05.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:05.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:05.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:05.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:05.056+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:05.352+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13409 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:05.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:05.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:05.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:05.467+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:06.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:06.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:06.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:06.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:06.307+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:06.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:06.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:06.069+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:06.070+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:06.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:06.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:06.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:06.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:06.515+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:07.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:07.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:07.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:07.056+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:07.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:07.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:07.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:07.291+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:07.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:07.555+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:07.555+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:07.555+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:07.556+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:08.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:08.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:08.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:08.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:08.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:08.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:08.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:08.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:08.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:08.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:08.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:08.533+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:08.534+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:09.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:09.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:09.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:09.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:09.132+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:09.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:09.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:09.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:09.292+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:09.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:09.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:09.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:09.484+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:09.485+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:10.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:10.353+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13503 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:10.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:10.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:10.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:10.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:10.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:10.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:10.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:10.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:10.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:10.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:10.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:10.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:10.483+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:11.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:11.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:11.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:11.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:11.270+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:11.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:11.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:11.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:11.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:11.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:11.192+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:11.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:11.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:11.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:11.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:11.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:12.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:12.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:12.190+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:12.191+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:12.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:12.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:12.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:12.296+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:12.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:12.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:12.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:12.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:13.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:13.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:13.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:13.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:13.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:13.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:13.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:13.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:13.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:13.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:13.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:13.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:13.464+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:14.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:14.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:14.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:14.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:14.288+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:14.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:14.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:14.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:14.152+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:14.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:14.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:14.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:14.494+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:15.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:15.355+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13593 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:15.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:15.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:15.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:15.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:15.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:15.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:15.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:15.199+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:15.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:15.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:15.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:15.467+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:16.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:16.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:16.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:16.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:16.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:16.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:16.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:16.211+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:16.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:16.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:16.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:16.474+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:17.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:17.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:17.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:17.281+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:17.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:17.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:17.239+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:17.240+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:17.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:17.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:17.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:17.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:17.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:17.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:17.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:17.433+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:18.477 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:18.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:18.477 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:18.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:18.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:18.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:18.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:18.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:18.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:18.478 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:18.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:18.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:18.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:18.474+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:18.475+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:19.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:19.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:19.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:19.296+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:19.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:19.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:19.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:19.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:19.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:19.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:19.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:20.437 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:20.356+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13687 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:20.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:20.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:20.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:20.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:20.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:20.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:20.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:20.323+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:20.438 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:20.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:20.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:20.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:20.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:20.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:20.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:20.159+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:20.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:20.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:20.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:20.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:20.437+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:21.396+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:21.396+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:21.396+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:21.397+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:21.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:21.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:21.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:21.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:21.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:22.420+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:22.420+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:22.420+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:22.421+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:22.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:22.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:22.400+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:22.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:22.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:22.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:22.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:22.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:22.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:22.193+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:23.416+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:23.416+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:23.416+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:23.417+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:23.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:23.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:23.197+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:23.198+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:23.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:23.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:23.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:23.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:24.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:24.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:24.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:24.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:24.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:24.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:24.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:24.153+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:24.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:24.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:24.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:24.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:24.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:24.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:24.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:24.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:24.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:24.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:24.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:24.407+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:25.408 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:25.358+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13780 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:25.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:25.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:25.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:25.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:25.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:25.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:25.409 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:25.372+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:25.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:25.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:25.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:25.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:25.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:25.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:25.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:25.144+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:25.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:25.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:25.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:25.408+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:26.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:26.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:26.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:26.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:26.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:26.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:26.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:26.382+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:26.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:26.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:26.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:26.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:27.381 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:27.382 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:27.382 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:27.382 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:27.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:27.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:27.130+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:27.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:27.130+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:27.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:27.130+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:27.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:27.131+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:27.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:27.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:27.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:27.381+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:27.382+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:28.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:28.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:28.339+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:28.174+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:28.174+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:28.174+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:28.174+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:28.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:28.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:28.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:28.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:28.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:29.342+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:29.342+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:29.342+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:29.343+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:29.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:29.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:29.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:29.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:29.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:29.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:29.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:29.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:29.144+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:30.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:30.359+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13874 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:30.350+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:30.350+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:30.350+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:30.351+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:30.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:30.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:30.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:30.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:30.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:30.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:30.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:30.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:30.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:30.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:30.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:30.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:31.391+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:31.391+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:31.391+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:31.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:31.391+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:31.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:31.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:31.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:31.177+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:31.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:31.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:31.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:31.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:32.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:32.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:32.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:32.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:32.212+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:32.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:32.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:32.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:32.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:32.416 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:32.254+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:32.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:32.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:32.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:32.414+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:33.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:33.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:33.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:33.422+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:33.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:33.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:33.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:33.175+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:33.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:33.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:33.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:33.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:33.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:33.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:33.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:33.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:34.439+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:34.439+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:34.439+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:34.439+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:34.242+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:34.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:34.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:34.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:34.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:34.209+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:35.450 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:35.360+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 13966 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:35.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:35.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:35.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:35.405+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:35.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:35.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:35.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:35.165+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:35.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:35.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:35.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:35.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:35.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:35.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:35.205+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:36.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:36.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:36.381 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:36.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:36.381 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:36.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:36.381 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:36.219+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:36.381 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:36.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:36.381 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:36.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:36.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:36.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:36.382 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:36.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:36.380+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:36.380+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:36.380+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:36.380+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:37.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:37.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:37.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:37.602 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:37.425+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:37.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:37.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:37.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:37.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:37.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:37.603 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:37.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:37.258+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:37.258+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:38.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:38.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:38.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:38.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:38.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:38.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:38.382+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:38.383+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:38.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:38.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:38.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:38.293+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:39.324 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:39.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:39.325 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:39.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:39.325 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:39.287+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:39.325 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:39.288+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:39.434+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:39.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:39.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:39.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:39.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:40.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:40.363+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14059 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:40.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:40.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:40.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:40.387+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:40.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:40.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:40.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:40.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:40.330+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:40.330+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:40.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:40.330+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:40.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:40.331+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:41.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:41.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:41.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:41.364+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:41.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:41.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:41.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:41.388+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:41.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:41.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:41.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:41.300+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:41.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:41.301+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:42.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:42.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:42.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:42.336+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:42.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:42.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:42.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:42.349+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:42.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:42.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:42.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:42.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:42.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:42.322+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:43.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:43.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:43.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:43.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:43.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:43.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:43.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:43.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:43.575 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:43.324+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:43.575 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:43.324+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:43.575 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:43.324+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:43.575 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:43.324+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:43.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:43.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:43.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:43.318+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:44.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:44.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:44.346+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:44.347+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:44.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:44.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:44.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:44.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:44.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:44.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:44.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:44.335+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:45.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:45.364+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14148 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:45.314+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:45.314+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:45.314+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:45.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:45.315+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:45.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:45.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:45.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:45.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:45.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:45.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:45.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:45.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:45.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:45.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:45.326+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:46.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:46.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:46.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:46.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:46.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:46.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:46.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:46.300+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:46.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:46.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:46.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:46.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:46.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:46.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:46.344+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:47.408+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:47.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:47.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:47.409+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:47.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:47.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:47.336+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:47.337+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:47.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:47.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:47.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:47.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:47.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:47.300+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:48.491 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:48.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:48.492 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:48.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:48.492 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:48.321+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:48.492 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:48.322+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:48.492 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:48.449+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:48.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:48.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:48.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:48.493 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:48.251+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:49.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:49.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:49.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:49.405+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:49.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:49.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:49.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:49.300+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:49.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:49.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:49.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:49.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:49.274+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:50.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:50.366+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14236 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:50.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:50.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:50.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:50.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:50.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:50.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:50.254+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:50.254+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:50.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:50.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:50.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:50.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:50.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:50.255+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:50.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:50.255+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:51.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:51.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:51.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:51.295+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:51.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:51.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:51.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:51.256+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:51.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:51.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:51.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:51.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:51.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:51.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:51.367+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:52.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:52.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:52.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:52.385+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:52.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:52.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:52.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:52.228+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:52.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:52.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:52.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:52.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:52.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:52.285+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:53.530 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:53.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:53.531 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:53.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:53.531 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:53.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:53.531 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:53.301+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:53.531 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:53.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:53.531 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:53.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:53.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:53.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:53.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:53.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:53.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:53.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:53.256+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:54.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:54.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:54.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:54.290+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:54.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:54.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:54.305+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:54.305+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:54.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:54.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:54.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:54.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T10:59:55.367+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14323 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T10:59:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:55.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:55.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:55.264+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:55.265+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:55.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:55.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:55.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:55.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:55.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:55.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:55.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:55.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:55.317+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:55.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:55.318+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:56.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:56.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:56.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:56.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:56.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:56.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:56.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:56.329+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:56.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:56.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:56.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:56.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:56.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:56.280+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:56.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:56.281+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:57.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:57.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:57.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:57.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:57.295+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:57.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:57.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:57.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:57.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:57.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:57.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:57.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:57.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:57.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:57.281+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:58.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:58.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:58.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:58.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:58.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:58.335+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:58.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:58.336+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:58.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:58.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:58.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:58.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:58.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:58.283+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:58.571 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:58.283+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:58.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:58.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:58.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:58.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:58.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:58.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:58.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:58.359+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:59.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:59.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:59.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T10:59:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T10:59:59.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:59.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:59.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:59.330+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T10:59:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T10:59:59.330+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T10:59:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:59.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:59.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:59.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:59.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:59.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T10:59:59.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 10:59:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T10:59:59.342+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:00.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:00.369+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14413 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:00.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:00.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:00.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:00.299+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:00.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:00.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:00.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:00.372+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:00.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:00.293+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:00.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:00.293+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:00.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:00.293+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:00.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:00.294+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:01.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:01.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:01.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:01.418+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:01.338+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:01.338+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:01.338+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:01.339+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:01.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:01.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:01.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:01.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:01.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:01.315+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:01.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:01.316+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:02.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:02.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:02.286+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:02.287+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:02.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:02.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:02.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:02.391+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:02.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:02.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:02.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:02.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:02.346+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:02.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:02.346+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:03.609 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:03.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:03.609 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:03.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:03.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:03.347+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:03.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:03.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:03.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:03.610 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:03.389+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:03.611 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:03.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:03.611 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:03.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:03.611 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:03.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:03.611 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:03.331+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:04.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:04.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:04.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:04.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:04.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:04.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:04.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:04.435+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:04.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:04.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:04.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:04.376+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:04.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:04.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:04.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:04.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:04.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:04.376+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:05.371+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14500 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:05.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:05.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:05.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:05.397+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:05.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:05.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:05.338+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:05.339+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:05.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:05.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:05.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:05.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:05.379+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:05.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:05.379+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:06.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:06.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:06.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:06.376+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:06.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:06.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:06.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:06.383+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:06.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:06.414+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:06.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:06.414+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:06.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:06.414+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:06.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:06.415+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:07.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:07.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:07.397+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:07.398+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:07.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:07.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:07.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:07.364+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:07.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:07.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:07.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:07.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:07.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:07.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:08.649 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:08.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:08.649 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:08.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:08.649 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:08.405+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:08.650 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:08.406+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:08.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:08.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:08.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:08.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:08.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:08.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:08.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:08.374+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:08.651 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:08.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:08.651 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:08.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:08.651 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:08.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:08.651 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:08.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T11:00:08.762 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-24T11:00:08.763 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T11:00:08.763 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:00:08.781 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-24T11:00:08.781 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:00:08.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T11:00:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:09.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:09.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:09.438+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:09.438+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:09.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:09.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:09.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:09.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:09.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:09.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:09.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:09.358+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:09.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:09.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:10.400 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:10.373+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14588 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:10.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:10.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:10.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:10.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:10.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:10.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:10.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:10.372+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:10.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:10.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:10.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:10.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:10.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:10.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:10.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:10.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:10.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:10.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:10.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:10.416+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:11.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:11.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:11.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:11.427+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:11.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:11.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:11.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:11.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:11.410+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:11.410+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:11.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:11.410+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:11.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:11.410+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:12.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:12.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:12.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:12.384+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:12.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:12.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:12.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:12.371+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:12.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:12.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:12.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:12.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:12.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:12.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:12.366+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:13.688 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:13.355+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:13.689 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:13.355+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:13.689 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:13.355+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:13.689 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:13.356+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:13.689 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:13.689 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:13.395+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:13.347+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:13.690 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:13.347+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:14.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:14.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:14.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:14.393+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:14.420+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:14.420+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:14.420+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:14.421+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:14.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:14.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:14.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:14.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:14.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:14.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:14.377+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:15.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:15.374+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14673 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:15.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:15.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:15.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:15.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:15.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:15.369+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:15.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:15.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:15.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:15.390+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:15.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:15.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:15.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:15.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:15.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:15.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:15.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:15.354+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:16.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:16.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:16.366+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:16.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:16.367+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:16.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:16.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:16.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:16.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:16.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:16.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:16.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:16.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:16.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:16.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:16.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:16.309+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:17.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:17.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:17.363+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:17.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:17.364+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:17.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:17.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:17.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:17.436+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:17.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:17.294+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:17.294+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:17.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:17.294+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:18.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:18.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:18.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:18.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:18.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:18.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:18.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:18.296+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:18.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:18.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:18.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:18.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:18.414+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:18.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:18.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:18.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:18.424+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:19.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:19.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:19.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:19.424+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:19.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:19.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:19.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:19.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:19.297+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:19.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:19.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:19.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:19.415+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:20.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:20.376+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14760 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:20.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:20.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:20.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:20.379+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:20.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:20.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:20.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:20.299+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:20.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:20.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:20.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:20.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:20.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:20.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:20.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:20.390+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:21.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:21.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:21.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:21.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:21.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:21.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:21.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:21.310+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:21.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:21.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:21.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:21.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:21.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:21.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:21.400+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:22.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:22.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:22.354+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:22.355+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:22.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:22.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:22.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:22.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:22.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:22.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:22.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:22.393+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:22.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:22.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:22.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:22.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:22.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:22.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:22.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:23.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:23.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:23.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:23.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:23.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:23.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:23.578 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:23.400+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:23.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:23.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:23.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:23.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:23.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:23.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:23.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:23.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:23.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:23.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:23.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:23.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:23.344+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:24.337+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:24.337+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:24.337+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:24.337+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:24.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:24.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:24.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:24.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:24.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:24.383+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:24.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:24.384+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:24.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:24.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:24.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:24.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:24.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:24.318+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:24.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:24.318+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:25.390 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:25.378+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14845 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:25.359+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:25.359+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:25.359+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:25.360+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:25.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:25.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:25.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:25.392 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:25.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:25.392 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:25.373+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:25.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:25.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:25.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:25.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:25.414+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:26.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:26.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:26.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:26.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:26.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:26.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:26.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:26.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:26.393+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:26.351+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:26.351+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:26.351+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:26.352+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:27.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:27.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:27.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:27.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:27.415+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:27.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:27.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:27.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:27.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:27.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:27.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:27.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:27.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:27.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:27.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:27.354+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:28.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:28.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:28.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:28.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:28.419+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:28.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:28.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:28.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:28.311+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:28.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:28.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:28.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:28.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:28.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:28.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:28.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:29.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:29.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:29.445+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:29.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:29.446+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:29.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:29.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:29.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:29.360+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:29.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:29.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:29.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:29.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:29.335+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:29.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:29.335+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:30.447 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:30.380+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 14932 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:30.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:30.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:30.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:30.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:30.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:30.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:30.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:30.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:30.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:30.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:30.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:30.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:30.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:30.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:30.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:30.345+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:30.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:30.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:30.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:30.448+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:31.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:31.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:31.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:31.391+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:31.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:31.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:31.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:31.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:31.455+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:31.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:31.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:31.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:31.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:31.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:31.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:31.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:31.310+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:32.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:32.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:32.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:32.429+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:32.430+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:32.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:32.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:32.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:32.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:32.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:32.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:33.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:33.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:33.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:33.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:33.578 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:33.400+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:33.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:33.401+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:33.579 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:33.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:33.579 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:33.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:33.579 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:33.386+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:33.579 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:33.386+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:33.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:33.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:33.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:33.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:33.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:33.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:33.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:33.241+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:34.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:34.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:34.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:34.281+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:34.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:34.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:34.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:34.367+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:34.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:34.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:34.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:34.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:34.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:34.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:34.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:34.342+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:35.505 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:35.381+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15017 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:35.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:35.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:35.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:35.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:35.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:35.342+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:35.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:35.342+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:35.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:35.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:35.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:35.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:35.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:35.373+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:35.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:35.374+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:35.507 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:35.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:35.508 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:35.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:35.508 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:35.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:35.508 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:35.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:36.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:36.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:36.361+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:36.361+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:36.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:36.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:36.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:36.251+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:36.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:36.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:36.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:36.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:36.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:36.351+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:36.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:36.351+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:37.376+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:37.376+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:37.376+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:37.377+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:37.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:37.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:37.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:37.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:37.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:37.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:37.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:37.229+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:37.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:37.605 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:37.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:37.605 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:37.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:37.605 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:37.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:38.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:38.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:38.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:38.227+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:38.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:38.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:38.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:38.427+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:38.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:38.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:38.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:38.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:38.325+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:39.284+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:39.284+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:39.284+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:39.285+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:39.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:39.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:39.426+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:39.426+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:39.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:39.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:39.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:39.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:39.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:39.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:40.407 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:40.382+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15105 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:40.407 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:40.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:40.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:40.392+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:40.392+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:40.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:40.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:40.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:40.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:40.409 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:40.271+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:40.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:40.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:40.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:40.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:40.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:40.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:40.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:40.277+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:41.356+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:41.356+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:41.356+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:41.356+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:41.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:41.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:41.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:41.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:41.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:41.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:41.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:41.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:41.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:41.253+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:42.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:42.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:42.390+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:42.390+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:42.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:42.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:42.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:42.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:42.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:42.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:42.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:42.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:42.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:42.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:42.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:42.303+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:43.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:43.415+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:43.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:43.415+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:43.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:43.415+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:43.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:43.415+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:43.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:43.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:43.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:43.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:43.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:43.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:43.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:43.353+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:43.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:43.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:43.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:43.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:43.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:43.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:43.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:43.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:44.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:44.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:44.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:44.453+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:44.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:44.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:44.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:44.333+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:44.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:44.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:44.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:44.306+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:45.384+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15191 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:45.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:45.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:45.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:45.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:45.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:45.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:45.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:45.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:45.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:45.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:45.313+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:45.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:45.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:45.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:45.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:45.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:45.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:45.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:45.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:46.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:46.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:46.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:46.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:46.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:46.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:46.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:46.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:46.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:46.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:46.499+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:46.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:46.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:46.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:46.308+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:46.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:46.308+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:47.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:47.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:47.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:47.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:47.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:47.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:47.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:47.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:47.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:47.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:47.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:47.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:47.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:47.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:47.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:47.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:47.297+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:47.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:47.297+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:48.530 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:48.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:48.530 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:48.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:48.530 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:48.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:48.530 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:48.377+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:48.530 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:48.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:48.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:48.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:48.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:48.269+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:48.531 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:48.270+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:48.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:48.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:48.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:48.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:48.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:49.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:49.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:49.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:49.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:49.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:49.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:49.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:49.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:49.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:49.553 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:49.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:49.838 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:49.551+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:49.838 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:49.551+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:49.838 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:49.551+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:49.839 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:49.553+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:50.409 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:50.387+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15279 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:50.410 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:50.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:50.410 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:50.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:50.410 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:50.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:50.410 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:50.214+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:50.410 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:50.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:50.411 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:50.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:50.411 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:50.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:50.411 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:50.328+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:50.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:50.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:50.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:50.512+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:51.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:51.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:51.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:51.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:51.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:51.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:51.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:51.166+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:51.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:51.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:51.489+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:51.490+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:52.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:52.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:52.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:52.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:52.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:52.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:52.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:52.194+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:52.781 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:52.531+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:52.781 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:52.531+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:52.782 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:52.531+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:52.782 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:52.532+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:53.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:53.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:53.198+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:53.199+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:53.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:53.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:53.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:53.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:53.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:53.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:53.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:53.491+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:54.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:54.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:54.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:54.383+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:54.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:54.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:54.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:54.161+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:54.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:54.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:54.529+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:54.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:55.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:00:55.388+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15364 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:00:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:55.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:55.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:55.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:55.382+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:55.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:55.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:55.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:55.173+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:55.853 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:55.570+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:55.854 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:55.570+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:55.854 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:55.570+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:55.854 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:55.572+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:56.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:56.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:56.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:56.187+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:56.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:56.188+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:56.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:56.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:56.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:56.395+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:56.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:56.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:56.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:56.564+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:57.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:57.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:57.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:57.395+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:57.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:57.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:57.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:57.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:57.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:57.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:57.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:57.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:57.517+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:58.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:58.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:58.391+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:58.392+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:58.160+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:58.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:58.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:58.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:58.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:58.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:58.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:58.926 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:58.514+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:59.344 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:59.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:59.344 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:59.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:59.344 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:59.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:00:59.344 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:00:59.171+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:59.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:59.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:59.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:00:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:00:59.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:00:59.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:59.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:59.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:59.527+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:00:59.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:00:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:00:59.528+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:00.402 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:00.390+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15452 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:00.403 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:00.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:00.403 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:00.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:00.403 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:00.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:00.403 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:00.145+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:00.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:00.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:00.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:00.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:00.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:00.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:00.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:00.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:00.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:00.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:00.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:00.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:00.504+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:01.341 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:01.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:01.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:01.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:01.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:01.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:01.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:01.103+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:01.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:01.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:01.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:01.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:01.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:02.341 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:02.341 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:02.341 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:02.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:02.149+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:02.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:02.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:02.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:02.509+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:02.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:02.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:02.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:02.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:03.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:03.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:03.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:03.333+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:03.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:03.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:03.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:03.182+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:03.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:03.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:03.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:03.504+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:04.686 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:04.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:04.740 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:04.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:04.741 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:04.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:04.741 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:04.464+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:04.741 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:04.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:04.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:04.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:04.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:04.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:04.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:04.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:04.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:04.743 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:04.221+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:05.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:05.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:05.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:05.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:05.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:05.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:05.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:05.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:05.394 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:05.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:05.394 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:05.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:05.394 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:05.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:05.394 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:05.172+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:05.392+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15538 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:05.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:05.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:05.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:05.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:05.478+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:06.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:06.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:06.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:06.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:06.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:06.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:06.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:06.287+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:06.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:06.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:06.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:06.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:06.439 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:06.181+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:06.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:06.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:06.437+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:06.438+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:07.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:07.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:07.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:07.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:07.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:07.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:07.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:07.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:07.478 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:07.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:07.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:07.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:07.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:07.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:07.479 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:07.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:07.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:07.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:07.477+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:07.478+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:08.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:08.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:08.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:08.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:08.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:08.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:08.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:08.254+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:08.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:08.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:08.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:08.512+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:09.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:09.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:09.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:09.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:09.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:09.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:09.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:09.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:09.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:09.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:09.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:09.524+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:09.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:09.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:10.406 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:10.394+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15625 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:10.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:10.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:10.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:10.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:10.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:10.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:10.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:10.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:10.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:10.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:10.157+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:10.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:10.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:10.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:10.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:10.570+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:11.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:11.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:11.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:11.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:11.118+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:11.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:11.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:11.549+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:11.550+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:12.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:12.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:12.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:12.159+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:12.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:12.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:12.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:12.312+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:12.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:12.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:12.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:12.560+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:13.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:13.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:13.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:13.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:13.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:13.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:13.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:13.179+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:13.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:13.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:13.559+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:13.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:13.559+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:14.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:14.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:14.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:14.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:14.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:14.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:14.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:14.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:14.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:14.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:14.537+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:14.537+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:15.521 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:15.396+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15709 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:15.521 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:15.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:15.521 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:15.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:15.521 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:15.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:15.522 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:15.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:15.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:15.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:15.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:15.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:15.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:15.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:15.522 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:15.320+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:15.536+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:15.536+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:15.536+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:15.537+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:16.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:16.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:16.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:16.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:16.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:16.316+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:16.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:16.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:16.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:16.540 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:16.204+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:16.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:16.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:16.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:16.540+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:17.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:17.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:17.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:17.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:17.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:17.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:17.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:17.566+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:18.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:18.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:18.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:18.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:18.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:18.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:18.597 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:18.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:18.597 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:18.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:18.597 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:18.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:18.597 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:18.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:18.597 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:18.211+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:18.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:18.595+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:18.595+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:18.595+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:18.596+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:19.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:19.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:19.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:19.200+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:19.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:19.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:19.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:19.229+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:19.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:19.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:19.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:19.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:19.612+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:20.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:20.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:20.399 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:20.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:20.399 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:20.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:20.399 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:20.190+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:20.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:20.399 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:20.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:20.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:20.249+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:20.400 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:20.250+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:20.398+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15797 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:20.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:20.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:20.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:20.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:20.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:20.642+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:21.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:21.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:21.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:21.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:21.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:21.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:21.639 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:21.203+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:21.639 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:21.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:21.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:21.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:21.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:21.227+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:21.640 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:21.227+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:21.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:21.953 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:21.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:21.953 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:21.638+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:21.954 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:21.639+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:22.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:22.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:22.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:22.235+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:22.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:22.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:22.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:22.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:22.594+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:22.594+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:22.594+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:22.595+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:23.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:23.228+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:23.581 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:23.228+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:23.581 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:23.228+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:23.581 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:23.228+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:23.581 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:23.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:23.581 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:23.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:23.582 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:23.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:23.582 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:23.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:23.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:23.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:23.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:23.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:23.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:23.604+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:24.561 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:24.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:24.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:24.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:24.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:24.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:24.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:24.209+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:24.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:24.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:24.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:24.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:24.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:24.236+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:24.563 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:24.236+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:24.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:24.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:24.561+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:24.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:24.562+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:25.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:25.399+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15882 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:25.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:25.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:25.196+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:25.196+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:25.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:25.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:25.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:25.222+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:25.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:25.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:25.586+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:25.586+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:26.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:26.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:26.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:26.219+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:26.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:26.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:26.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:26.194+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:26.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:26.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:26.622+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:26.622+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:27.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:27.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:27.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:27.173+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:27.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:27.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:27.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:27.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:27.640+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:27.640+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:27.640+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:27.640+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:28.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:28.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:28.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:28.215+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:28.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:28.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:28.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:28.188+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:28.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:28.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:28.676+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:28.677+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:29.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:29.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:29.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:29.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:29.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:29.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:29.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:29.208+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:30.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:29.707+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:30.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:29.707+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:30.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:29.707+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:30.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:29.708+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:30.412 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:30.401+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 15970 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:30.413 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:30.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:30.413 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:30.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:30.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:30.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:30.414 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:30.184+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:30.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:30.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:30.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:30.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:30.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:30.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:30.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:30.191+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:31.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:30.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:30.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:30.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:30.722+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:31.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:31.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:31.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:31.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:31.159+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:31.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:31.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:31.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:31.144+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:31.674+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:31.674+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:31.674+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:31.675+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:32.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:32.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:32.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:32.179+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:32.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:32.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:32.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:32.111+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:33.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:32.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:32.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:32.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:32.720+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:33.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:33.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:33.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:33.109+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:33.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:33.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:33.157+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:33.157+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:33.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:33.764+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:34.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:34.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:34.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:34.108+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:34.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:34.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:34.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:34.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:35.120 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:34.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:35.120 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:34.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:35.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:34.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:35.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:34.724+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:35.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:35.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:35.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:35.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:35.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:35.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:35.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:35.120+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:35.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:35.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:35.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:35.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:35.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:35.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:35.405 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:35.133+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:35.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:35.403+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16055 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:36.081 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:35.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:36.081 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:35.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:36.081 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:35.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:36.082 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:35.721+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:36.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:36.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:36.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:36.081+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:36.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:36.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:36.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:36.171+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:37.048 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:36.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:37.048 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:36.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:37.048 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:36.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:37.048 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:36.737+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:37.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:37.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:37.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:37.048+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:37.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:37.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:37.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:37.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:38.132 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.006+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:37.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:37.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:37.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:37.764+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:38.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:38.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:38.132+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:38.132+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:38.996 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:38.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.996 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:38.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.996 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:38.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:38.996 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:38.746+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:39.360 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.995+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:39.360 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.995+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:39.361 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.995+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:39.361 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:38.996+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:39.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:39.112+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:39.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:39.112+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:39.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:39.112+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:39.362 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:39.112+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:40.016 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:39.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:40.016 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:39.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:40.016 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:39.705+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:40.016 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:39.706+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:40.406 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:40.406 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:40.406 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:40.406 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.016+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:40.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:40.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:40.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:40.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:40.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:40.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:40.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:40.068+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:40.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:40.405+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16143 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:41.057 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:40.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:41.057 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:40.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:41.057 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:40.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:41.057 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:40.757+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:41.057 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.967+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:41.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.967+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:41.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.967+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:41.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:40.968+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:41.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:41.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:41.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:41.056+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:41.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:41.057+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:41.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:41.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:41.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:41.758+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:41.996+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:42.091 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:41.996+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:42.092 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:41.996+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:42.092 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:41.997+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:42.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:42.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:42.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:42.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:42.091+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:43.028 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:42.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.028 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:42.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.028 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:42.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.028 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:42.772+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:43.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:43.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:43.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:43.103+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:43.103+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:43.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:43.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:43.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:43.028+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:43.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:43.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:43.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:43.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:43.784+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:44.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.002+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:44.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:44.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:44.087+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:44.087+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:45.133 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:45.133 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:44.973+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:44.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:44.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:44.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:44.819+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:45.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:45.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:45.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:45.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:45.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:45.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:45.408 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:45.133+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:45.407+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16229 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:46.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:45.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:46.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:45.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:45.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:45.827+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:45.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:45.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:45.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:46.128 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:45.998+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:46.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:46.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:46.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:46.127+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:47.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:46.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:46.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:46.832+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:46.832+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:47.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:47.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.003+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:47.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:47.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:47.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:47.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:47.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:47.158+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:48.161 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:48.161 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:48.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:48.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:47.985+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:48.162 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:47.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:48.162 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:47.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:48.162 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:47.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:48.163 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:47.851+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:48.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:48.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:48.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:48.161+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:49.166 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.015+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:48.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:48.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:49.167 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:48.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:49.168 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:48.853+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:49.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:49.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:49.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:49.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:49.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:49.166+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:49.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:49.166+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:50.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:49.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:49.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:49.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:49.901+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:50.159 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:50.160 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:49.980+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:50.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:50.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:50.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:50.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:50.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:50.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:50.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:50.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:50.407+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16317 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:51.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:50.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:51.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:50.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:50.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:50.872+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:51.004+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:51.004+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:51.004+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:51.150 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:51.005+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:51.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:51.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:51.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:51.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:51.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:51.184+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:52.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:52.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:52.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:52.024+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:51.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:51.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:51.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:51.890+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:52.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:52.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:52.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:52.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:52.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:52.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:52.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:52.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:52.892+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:53.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:53.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:53.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:53.010+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:53.583 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:53.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:53.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:53.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:53.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:53.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:53.584 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:53.280+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:53.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:53.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:53.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:53.885+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:54.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:54.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:54.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:54.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:54.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:54.017+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:54.489 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:54.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:54.489 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:54.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:54.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:54.250+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:54.490 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:54.250+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:54.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:54.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:54.886+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:54.886+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:55.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:55.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:55.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:55.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:55.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:55.534 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:01:55.409+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16402 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:01:55.534 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:55.217+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:55.534 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:55.217+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:55.535 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:55.217+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:55.535 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:55.217+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:55.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:55.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:55.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:55.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:56.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:56.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:56.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:56.036+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:56.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:56.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:56.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:56.202+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:57.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:57.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:57.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:57.295 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.004+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:57.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:56.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:57.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:56.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:57.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:56.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:57.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:56.968+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:57.296 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:57.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:57.296 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:57.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:57.296 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:57.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:57.296 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:57.182+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:57.962+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:58.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:58.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:58.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:58.015+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:58.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:58.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:58.160+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:58.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:58.160+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:58.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:58.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:58.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:01:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:01:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:01:58.971+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:59.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:59.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:59.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:01:59.030+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:59.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:59.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:59.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:59.205+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:01:59.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:01:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:01:59.206+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:00.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:00.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:00.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:00.366 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.011+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:00.366 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:00.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:00.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:00.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:00.017+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:00.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:00.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:00.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:00.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:00.201+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:00.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:00.412+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16490 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:01.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:01.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:01.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:01.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:01.034+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:01.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:01.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:01.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:01.222+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.975+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.976+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.976+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:00.976+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:02.243 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:02.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:02.243 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:02.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:02.243 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:02.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:02.243 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:02.064+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:02.244 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:01.982+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:02.244 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:01.982+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:02.244 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:01.982+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:02.244 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:01.983+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:02.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:02.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:02.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:02.242+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:02.243+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:03.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:03.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:03.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:03.245+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:03.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:03.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:03.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:03.083+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:03.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:03.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:03.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:03.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:03.030+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:04.101 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:04.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:04.101 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:04.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:04.102 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:04.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:04.102 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:04.028+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:04.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:04.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:04.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:04.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:04.101+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:04.285+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:04.285+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:04.285+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:04.286+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:05.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:05.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:05.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:05.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:05.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:05.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:05.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:05.060+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:05.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:05.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:05.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:05.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:05.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:05.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:05.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:05.058+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:05.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:05.414+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16576 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:05.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:05.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:05.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:05.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:05.313+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:06.344 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:06.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:06.345 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:06.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:06.345 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:06.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:06.345 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:06.105+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:06.346 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:06.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:06.346 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:06.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:06.346 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:06.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:06.346 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:06.063+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:06.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:06.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:06.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:06.344+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:07.311+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:07.311+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:07.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:07.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:07.146+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:07.147+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:07.147+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:07.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:07.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:07.110+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:07.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:07.110+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:08.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:08.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:08.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:08.139+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:08.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:08.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:08.307+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:08.307+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:08.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:08.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:08.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:08.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T11:02:08.912 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-24T11:02:08.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:02:08.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-24T11:02:08.963 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip: Cannot mkdir: Input/output error 2024-03-24T11:02:08.964 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-dstaddr: Cannot open: Input/output error 2024-03-24T11:02:08.964 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-nexthdr: Cannot open: Input/output error 2024-03-24T11:02:08.964 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-protcol: Cannot open: Input/output error 2024-03-24T11:02:08.964 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-checksum: Cannot open: Input/output error 2024-03-24T11:02:08.964 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-version: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-ihl: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-fragment: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-totlen: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-version: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-plen: Cannot open: Input/output error 2024-03-24T11:02:08.965 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-dstaddr: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast: Cannot mkdir: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation: Cannot mkdir: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope01: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope02: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope03: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.966 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope04: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope03: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope01: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope02: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope04: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood: Cannot mkdir: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld01: Cannot open: Input/output error 2024-03-24T11:02:08.967 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld03: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld05: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld02: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld05: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld01: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld02: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.968 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld06: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld03: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld04: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld06: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld04: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood: Cannot mkdir: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld02: Cannot open: Input/output error 2024-03-24T11:02:08.969 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld01: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld02: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld01: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route: Cannot mkdir: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-redirect: Cannot open: Input/output error 2024-03-24T11:02:08.970 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-if: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-gw: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-ifdown: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-if: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-dst: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-dst: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-gw: Cannot open: Input/output error 2024-03-24T11:02:08.971 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-ifdown: Cannot open: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-rmmod: Cannot open: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-redirect: Cannot open: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-rmmod: Cannot open: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp: Cannot mkdir: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff: Cannot mkdir: Input/output error 2024-03-24T11:02:08.972 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff14: Cannot open: Input/output error 2024-03-24T11:02:08.973 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff06: Cannot open: Input/output error 2024-03-24T11:02:08.973 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff09: Cannot open: Input/output error 2024-03-24T11:02:08.973 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff02: Cannot open: Input/output error 2024-03-24T11:02:08.973 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff13: Cannot open: Input/output error 2024-03-24T11:02:08.973 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff01: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff09: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff03: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff04: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff10: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff05: Cannot open: Input/output error 2024-03-24T11:02:08.974 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff04: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff07: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff11: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff12: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff05: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff10: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff07: Cannot open: Input/output error 2024-03-24T11:02:08.975 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff11: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff12: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff13: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff08: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff14: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff03: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff02: Cannot open: Input/output error 2024-03-24T11:02:08.976 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff08: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff06: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff01: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport: Cannot mkdir: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport04: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport03: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport12: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport07: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport05: Cannot open: Input/output error 2024-03-24T11:02:08.977 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport04: Cannot open: Input/output error 2024-03-24T11:02:08.978 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport14: Cannot open: Input/output error 2024-03-24T11:02:08.978 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport10: Cannot open: Input/output error 2024-03-24T11:02:08.978 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport02: Cannot open: Input/output error 2024-03-24T11:02:08.978 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport05: Cannot open: Input/output error 2024-03-24T11:02:08.978 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport06: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport01: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport09: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport13: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport10: Cannot open: Input/output error 2024-03-24T11:02:08.979 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport07: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport02: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport08: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport12: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport01: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport08: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport03: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport11: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport13: Cannot open: Input/output error 2024-03-24T11:02:08.980 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport11: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport14: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport06: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport09: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic: Cannot mkdir: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic04: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic06: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic13: Cannot open: Input/output error 2024-03-24T11:02:08.981 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic14: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic06: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic11: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic05: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic01: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic12: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic11: Cannot open: Input/output error 2024-03-24T11:02:08.982 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic10: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic07: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic10: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic08: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic13: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic03: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic08: Cannot open: Input/output error 2024-03-24T11:02:08.983 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic02: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic09: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic03: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic01: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic04: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic07: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic09: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic02: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic12: Cannot open: Input/output error 2024-03-24T11:02:08.984 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic14: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic05: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport: Cannot mkdir: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport07: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport14: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport11: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport06: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport09: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport02: Cannot open: Input/output error 2024-03-24T11:02:08.985 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport04: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport01: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport08: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport12: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport09: Cannot open: Input/output error 2024-03-24T11:02:08.986 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport05: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport13: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport05: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport08: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport02: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport14: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport12: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport11: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport10: Cannot open: Input/output error 2024-03-24T11:02:08.987 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport03: Cannot open: Input/output error 2024-03-24T11:02:08.988 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport04: Cannot open: Input/output error 2024-03-24T11:02:08.988 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport13: Cannot open: Input/output error 2024-03-24T11:02:08.988 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport06: Cannot open: Input/output error 2024-03-24T11:02:08.988 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport07: Cannot open: Input/output error 2024-03-24T11:02:08.988 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport10: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport01: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport03: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend: Cannot mkdir: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend12: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend09: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend03: Cannot open: Input/output error 2024-03-24T11:02:08.989 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend06: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend11: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend06: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend07: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend04: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend11: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend01: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend10: Cannot open: Input/output error 2024-03-24T11:02:08.990 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend02: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend04: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend14: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend13: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend12: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend01: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend05: Cannot open: Input/output error 2024-03-24T11:02:08.991 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend14: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend02: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend07: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend08: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend03: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend05: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend10: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend08: Cannot open: Input/output error 2024-03-24T11:02:08.992 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend13: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend09: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic: Cannot mkdir: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic04: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic05: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic09: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic01: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic02: Cannot open: Input/output error 2024-03-24T11:02:08.993 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic07: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic03: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic03: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic04: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic01: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic05: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic14: Cannot open: Input/output error 2024-03-24T11:02:08.994 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic11: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic12: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic06: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic13: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic08: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic02: Cannot open: Input/output error 2024-03-24T11:02:08.995 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic14: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic06: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic11: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic09: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic07: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic10: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic13: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic08: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic12: Cannot open: Input/output error 2024-03-24T11:02:08.996 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic10: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup: Cannot mkdir: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup09: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup06: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup13: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup10: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup09: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup03: Cannot open: Input/output error 2024-03-24T11:02:08.997 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup01: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup03: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup02: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup11: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup01: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup10: Cannot open: Input/output error 2024-03-24T11:02:08.998 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup14: Cannot open: Input/output error 2024-03-24T11:02:08.999 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/Makefile: Cannot open: Input/output error 2024-03-24T11:02:08.999 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup08: Cannot open: Input/output error 2024-03-24T11:02:09.000 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.000 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup05: Cannot open: Input/output error 2024-03-24T11:02:09.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi026.front.sepia.ceph.com 2024-03-24T11:02:09.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:02:09.001 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T11:02:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-24T11:02:09.035 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup04: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup13: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup07: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup06: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup12: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup04: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup05: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup12: Cannot open: Input/output error 2024-03-24T11:02:09.036 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup11: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup02: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup14: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup08: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup07: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff: Cannot mkdir: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff14: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff02: Cannot open: Input/output error 2024-03-24T11:02:09.037 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff06: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff02: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff06: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff09: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff08: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff11: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff10: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff11: Cannot open: Input/output error 2024-03-24T11:02:09.038 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff09: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff10: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff13: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff05: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff07: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff13: Cannot open: Input/output error 2024-03-24T11:02:09.039 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff01: Cannot open: Input/output error 2024-03-24T11:02:09.040 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff14: Cannot open: Input/output error 2024-03-24T11:02:09.040 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff01: Cannot open: Input/output error 2024-03-24T11:02:09.040 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff12: Cannot open: Input/output error 2024-03-24T11:02:09.040 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff03: Cannot open: Input/output error 2024-03-24T11:02:09.040 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff04: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff05: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff03: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff07: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff08: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff12: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff04: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip: Cannot mkdir: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip09: Cannot open: Input/output error 2024-03-24T11:02:09.041 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip08: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip06: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip02: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip01: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip04: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip12: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip03: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip03: Cannot open: Input/output error 2024-03-24T11:02:09.042 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip13: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip14: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip02: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip11: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip04: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.043 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip06: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip10: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip12: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip09: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip14: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip07: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip08: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip13: Cannot open: Input/output error 2024-03-24T11:02:09.044 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip05: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip10: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip07: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip01: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip11: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip05: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso: Cannot mkdir: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso11: Cannot open: Input/output error 2024-03-24T11:02:09.045 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso07: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso14: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso05: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso08: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso04: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso10: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso06: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso08: Cannot open: Input/output error 2024-03-24T11:02:09.046 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso12: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso03: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso01: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso06: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso13: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso07: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso02: Cannot open: Input/output error 2024-03-24T11:02:09.047 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso04: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso12: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso01: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso10: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso09: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso14: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso02: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso09: Cannot open: Input/output error 2024-03-24T11:02:09.048 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso11: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso03: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso05: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso13: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale: Cannot mkdir: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale07: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale01: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale13: Cannot open: Input/output error 2024-03-24T11:02:09.049 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale06: Cannot open: Input/output error 2024-03-24T11:02:09.050 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale12: Cannot open: Input/output error 2024-03-24T11:02:09.050 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale05: Cannot open: Input/output error 2024-03-24T11:02:09.050 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale11: Cannot open: Input/output error 2024-03-24T11:02:09.050 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale02: Cannot open: Input/output error 2024-03-24T11:02:09.050 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale02: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale07: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale01: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale11: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale14: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.051 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale13: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale08: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale06: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale10: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale05: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale04: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale10: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale09: Cannot open: Input/output error 2024-03-24T11:02:09.052 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale09: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale03: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale03: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale04: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale08: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale12: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale14: Cannot open: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns: Cannot mkdir: Input/output error 2024-03-24T11:02:09.053 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns6-stress: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns4-stress: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress01-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress02-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp: Cannot mkdir: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-download-stress: Cannot open: Input/output error 2024-03-24T11:02:09.054 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress01-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress02-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-upload-stress: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress02-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-upload-stress: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress01-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.055 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-download-stress: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/README: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface: Cannot mkdir: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-updown: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-updown: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-addlarge: Cannot open: Input/output error 2024-03-24T11:02:09.056 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-adddel: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-addr-change: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-addlarge: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-addlarge: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-mtu-change: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-addlarge: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-mtu-change: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-adddel: Cannot open: Input/output error 2024-03-24T11:02:09.057 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-adddel: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-adddel: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp: Cannot mkdir: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport: Cannot mkdir: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport03: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport05: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport02: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport05: Cannot open: Input/output error 2024-03-24T11:02:09.058 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport01: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport07: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport06: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport02: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport06: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport03: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport04: Cannot open: Input/output error 2024-03-24T11:02:09.059 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport04: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport01: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport07: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic: Cannot mkdir: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic05: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic04: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic06: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic03: Cannot open: Input/output error 2024-03-24T11:02:09.060 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.061 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.061 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic02: Cannot open: Input/output error 2024-03-24T11:02:09.061 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic06: Cannot open: Input/output error 2024-03-24T11:02:09.061 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic01: Cannot open: Input/output error 2024-03-24T11:02:09.061 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic07: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic07: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic05: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic02: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic03: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic01: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic04: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.062 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic: Cannot mkdir: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic01: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic02: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic06: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic04: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic07: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic01: Cannot open: Input/output error 2024-03-24T11:02:09.063 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic03: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic05: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic02: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic04: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic03: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic06: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic05: Cannot open: Input/output error 2024-03-24T11:02:09.064 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic07: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip: Cannot mkdir: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip02: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip06: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip04: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip05: Cannot open: Input/output error 2024-03-24T11:02:09.065 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip06: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip07: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip01: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip03: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip02: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip01: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip05: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip03: Cannot open: Input/output error 2024-03-24T11:02:09.066 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip07: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip04: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http: Cannot mkdir: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress01-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http4-stress: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress02-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http6-stress: Cannot open: Input/output error 2024-03-24T11:02:09.067 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh: Cannot mkdir: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress02-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh6-stress: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh4-stress: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/00_Descriptions.txt: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress03-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress01-rmt: Cannot open: Input/output error 2024-03-24T11:02:09.068 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/stress: Cannot utime: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4: Cannot mkdir: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl: Cannot mkdir: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/random_gen.py: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/runtest: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/setacl_stress.py: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_long_acl.py: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleanusers.py: Cannot open: Input/output error 2024-03-24T11:02:09.069 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/README: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/create_users.py: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_acl.py: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/acl1.c: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleangroups.py: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks: Cannot mkdir: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/netsync.c: Cannot open: Input/output error 2024-03-24T11:02:09.070 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy_info: Cannot open: Input/output error 2024-03-24T11:02:09.071 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.h: Cannot open: Input/output error 2024-03-24T11:02:09.071 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.071 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.py: Cannot open: Input/output error 2024-03-24T11:02:09.071 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.c: Cannot open: Input/output error 2024-03-24T11:02:09.071 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/README: Cannot open: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy: Cannot mkdir: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy/locktests.tar.gz: Cannot open: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/VERSION: Cannot open: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iproute: Cannot mkdir: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iproute/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iproute/ip_tests.sh: Cannot open: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iptables: Cannot mkdir: Input/output error 2024-03-24T11:02:09.072 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iptables/iptables_tests.sh: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/iptables/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp: Cannot mkdir: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib: Cannot mkdir: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/addrs.c: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/connectx.c: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/bindx.c: Cannot open: Input/output error 2024-03-24T11:02:09.073 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/opt_info.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/peeloff.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/sendmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/recvmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests: Cannot mkdir: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_inaddr_any.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sctp_sendrecvmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.074 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_autoclose.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_basic.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_send.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_threads.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_getname.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_peeloff.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connectx.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_socket_bind_listen.c: Cannot open: Input/output error 2024-03-24T11:02:09.075 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_shutdown.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_accept_close.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_nonblock.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvfrom.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connect.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_initmsg_connect.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.076 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendto.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connect.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_tcp_style.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sockopt.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connectx.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_fragments.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_timetolive.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_recvmsg.c: Cannot open: Input/output error 2024-03-24T11:02:09.077 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_shutdown.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_events.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sockopt.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_abort.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_rtoinfo.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_addrs.c: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib: Cannot mkdir: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.h: Cannot open: Input/output error 2024-03-24T11:02:09.078 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.c: Cannot open: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/include: Cannot mkdir: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet: Cannot mkdir: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet/sctp.h: Cannot open: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs: Cannot mkdir: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux: Cannot mkdir: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.079 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/fsx.sh: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress: Cannot mkdir: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/make_tree.c: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/nfsstress: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04: Cannot mkdir: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04_create_file.c: Cannot open: Input/output error 2024-03-24T11:02:09.080 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01: Cannot mkdir: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.c: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.h: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfslock01: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_dgen.c: Cannot open: Input/output error 2024-03-24T11:02:09.081 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_frk.c: Cannot open: Input/output error 2024-03-24T11:02:09.082 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_func.c: Cannot open: Input/output error 2024-03-24T11:02:09.082 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.082 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01: Cannot mkdir: Input/output error 2024-03-24T11:02:09.082 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.082 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/nfsstat01: Cannot open: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02: Cannot mkdir: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/nfs02: Cannot open: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01: Cannot mkdir: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/nfs01: Cannot open: Input/output error 2024-03-24T11:02:09.083 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/open_files.c: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03: Cannot mkdir: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/nfs03: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/README: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/CTHON04_README: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress: Cannot mkdir: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/global.h: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/nfs_fsstress.sh: Cannot open: Input/output error 2024-03-24T11:02:09.084 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/xfscompat.h: Cannot open: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/fsstress.c: Cannot open: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/network: Cannot utime: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21: Cannot mkdir: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/COPYING: Cannot open: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/debug.c: Cannot open: Input/output error 2024-03-24T11:02:09.085 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts: Cannot mkdir: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00check_cdrom_presence: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/mem_alloc: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ramsnake: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ltp: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memxfer5b: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memtest: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/cpufreq: Cannot open: Input/output error 2024-03-24T11:02:09.086 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00xbonkers: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/coords: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00checklatest: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/build_kernel: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/schedutils: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/time_test: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/lame: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ipmitool: Cannot open: Input/output error 2024-03-24T11:02:09.087 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/bonnie++: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/random_syscall: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/nasm: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/QUICKSTART: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/CHANGELOG: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/nfs_logging: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/Makefile: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/test-tests.tar.gz: Cannot open: Input/output error 2024-03-24T11:02:09.088 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/fast-tests.tar.gz: Cannot open: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/run.c: Cannot open: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/README: Cannot open: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/Install: Cannot open: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:gzip: stdin: Stale file handle 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases/pounder21/fancy_timed_loop.c: Cannot open: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: Child returned status 1 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-03-24T11:02:09.089 INFO:tasks.workunit.client.0.smithi026.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-24T11:02:09.090 INFO:tasks.workunit.client.0.smithi026.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:02:09.090 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-24T11:02:09.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T11:02:09.111 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-24T11:02:09.111 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:02:09.188 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:09.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:09.188 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:09.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:09.188 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:09.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:09.188 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:09.060+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:09.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:09.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:09.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:09.188+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:09.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:09.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:09.340+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:09.341+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:10.326 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:10.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:10.327 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:10.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:10.327 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:10.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:10.327 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:10.199+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:10.327 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:10.038+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:10.327 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:10.038+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:10.328 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:10.038+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:10.328 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:10.040+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:10.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:10.416+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16664 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:10.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:10.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:10.326+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:10.327+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:11.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:11.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:11.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:11.281+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:11.281+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:11.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:11.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:11.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:11.161+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:11.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:11.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:11.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:11.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:11.080+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:12.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:12.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:12.270+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:12.271+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:12.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:12.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:12.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:12.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:12.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:12.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:12.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:12.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:12.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:12.049+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:13.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:13.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:13.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:13.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:13.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:13.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:13.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:13.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:13.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:13.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:13.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:13.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:13.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:13.187+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:13.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:13.187+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:14.183 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:14.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:14.183 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:14.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:14.183 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:14.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:14.184 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:14.097+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:14.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:14.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:14.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:14.257+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:14.183+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:14.183+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:14.183+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:14.183+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:15.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:15.181+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:15.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:15.181+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:15.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:15.181+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:15.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:15.182+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:15.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:15.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:15.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:15.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:15.144+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:15.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:15.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:15.420 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:15.276+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:15.421 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:15.276+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:15.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:15.418+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16751 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:16.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:16.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:16.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:16.108+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:16.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:16.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:16.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:16.293+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:16.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:16.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:16.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:16.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:16.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:16.231+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:17.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:17.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:17.197+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:17.197+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:17.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:17.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:17.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:17.086+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:17.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:17.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:17.252+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:17.253+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:18.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:18.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:18.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:18.201+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:18.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:18.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:18.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:18.069+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:18.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:18.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:18.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:18.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:18.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:18.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:18.303+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:19.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:19.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:19.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:19.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:19.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:19.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:19.313 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:19.044+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:19.313 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:19.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:19.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:19.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:19.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:19.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:19.314 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:19.165+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:19.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:19.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:19.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:19.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:19.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:19.312+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:19.684 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:19.313+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:20.329 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:20.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:20.330 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:20.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:20.330 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:20.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:20.330 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:20.079+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:20.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:20.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:20.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:20.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:20.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:20.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:20.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:20.200+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:20.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:20.420+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16840 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:20.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:20.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:20.329+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:20.329+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:21.376 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:21.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:21.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:21.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:21.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:21.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:21.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:21.108+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:21.377 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:21.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:21.378 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:21.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:21.378 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:21.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:21.378 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:21.223+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:21.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:21.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:21.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:21.376+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:22.119 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:22.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:22.119 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:22.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:22.120 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:22.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:22.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:22.119+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:22.415 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:22.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:22.415 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:22.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:22.415 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:22.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:22.415 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:22.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:22.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:22.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:22.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:22.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:22.414+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:23.412 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:23.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:23.412 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:23.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:23.412 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:23.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:23.412 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:23.199+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:23.413 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:23.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:23.413 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:23.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:23.413 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:23.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:23.413 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:23.116+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:23.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:23.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:23.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:23.412+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:24.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:24.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:24.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:24.079+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:24.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:24.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:24.224+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:24.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:24.225+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:24.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:24.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:24.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:24.450+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:25.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:25.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:25.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:25.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:25.244+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:25.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:25.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:25.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:25.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:25.424 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:25.083+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:25.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:25.422+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 16927 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:25.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:25.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:25.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:25.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:25.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:25.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:25.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:25.443+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:26.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:26.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:26.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:26.228+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:26.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:26.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:26.089+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:26.090+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:26.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:26.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:26.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:26.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:26.468+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:27.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:27.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:27.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:27.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:27.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:27.258+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:27.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:27.259+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:27.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:27.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:27.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:27.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:27.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:27.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:27.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:27.048+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:27.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:27.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:27.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:27.443+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:28.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:28.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:28.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:28.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:28.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:28.243+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:28.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:28.243+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:28.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:28.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:28.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:28.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:28.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:28.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:28.398 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:28.015+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:28.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:28.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:28.396+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:28.397+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:29.369 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:29.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:29.369 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:29.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:29.369 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:29.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:29.369 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:29.287+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:29.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:29.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:29.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:29.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:29.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:29.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:29.370 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:29.045+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:29.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:29.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:29.368+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:29.369+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:30.305 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:30.040+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:30.305 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:30.040+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:30.305 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:30.040+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:30.305 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:30.041+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:30.423+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17013 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:30.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:30.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:30.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:30.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:30.305+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:30.392+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:30.392+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:30.392+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:30.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:30.392+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:31.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:31.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:31.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:31.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:31.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:31.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:31.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:31.076+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:31.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:31.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:31.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:31.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:31.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:31.301+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:31.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:31.301+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:31.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:31.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:31.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:31.407+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:32.343+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:32.343+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:32.343+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:32.343+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:32.357+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:32.358+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:32.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:32.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:32.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:32.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:32.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:32.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:32.124+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:33.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:33.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:33.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:33.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:33.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:33.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:33.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:33.322+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:33.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:33.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:33.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:33.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:33.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:33.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:33.351 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:33.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:33.349+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:33.349+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:33.349+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:33.350+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:34.157 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:34.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:34.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:34.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:34.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:34.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:34.158 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:34.047+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:34.362+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:34.362+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:34.362+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:34.364+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:34.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:34.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:34.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:34.333+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:35.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:35.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:35.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:35.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:35.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:35.306+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:35.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:35.306+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:35.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:35.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:35.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:35.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:35.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:35.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:35.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:35.060+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:35.425+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17098 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:35.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:35.407+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:36.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:36.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:36.356 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:36.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:36.357 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:36.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:36.357 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:36.020+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:36.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:36.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:36.356+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:36.356+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:36.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:36.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:36.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:36.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:36.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:36.368+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:36.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:36.369+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:37.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:37.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:37.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:37.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:37.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:37.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:37.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:37.044+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:37.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:37.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:37.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:37.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:37.400+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:37.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:37.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:37.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:37.389+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:38.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:38.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:38.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:38.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:38.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:38.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:38.426 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:38.076+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:38.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:38.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:38.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:38.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:38.426+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:38.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:38.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:38.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:38.438+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:39.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:39.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:39.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:39.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:39.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:39.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:39.404 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:39.033+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:39.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:39.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:39.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:39.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:39.403+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:39.440+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:39.440+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:39.440+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:39.441+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:40.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:40.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:40.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:40.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:40.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:40.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:40.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:40.065+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:40.428 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:40.428 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:40.428 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:40.428 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:40.408+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:40.427+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17187 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:40.455+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:40.455+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:40.455+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:40.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:41.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:41.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:41.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:41.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:41.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:41.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:41.423 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:41.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:41.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:41.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:41.443+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:41.444+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:41.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:41.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:41.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:41.422+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:42.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:42.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:42.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:42.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:42.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:42.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:42.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:42.036+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:42.472+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:42.472+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:42.472+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:42.473+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:42.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:42.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:42.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:42.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:43.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:43.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:43.353 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:43.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:43.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:43.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:43.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:43.004+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:43.504+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:43.504+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:43.504+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:43.504+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:43.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:43.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:43.427+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:43.428+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:44.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:44.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:44.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:44.036+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:44.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:44.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:44.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:44.525+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:44.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:44.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:44.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:44.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:44.422+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:45.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:45.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:45.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:45.429+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17272 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:45.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:45.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:45.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:45.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:45.469+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:45.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:45.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:45.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:45.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:46.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:46.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:46.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:46.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:46.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:46.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:46.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:46.088+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:46.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:46.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:46.463+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:46.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:46.464+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:46.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:46.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:46.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:46.540+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:47.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:47.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:47.113+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:47.114+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:47.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:47.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:47.508+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:47.508+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:47.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:47.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:47.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:47.494+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:48.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:48.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:48.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:48.119+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:48.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:48.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:48.481+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:48.481+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:48.485+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:48.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:48.485+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:48.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:48.485+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:48.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:48.485+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:49.072+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:49.072+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:49.072+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:49.073+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:49.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:49.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:49.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:49.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:49.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:49.454+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:49.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:49.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:49.457+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:49.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:49.458+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:50.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:50.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:50.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:50.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:50.028+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:50.431+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17358 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:50.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:50.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:50.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:50.466+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:50.467+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:50.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:50.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:50.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:50.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:50.484+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:51.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:51.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:51.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:51.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:51.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:51.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:51.439 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:51.016+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:51.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:51.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:51.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:51.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:51.438+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:51.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:51.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:51.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:51.516+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:51.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:51.517+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:52.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:52.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:52.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:52.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:52.036+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:52.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:52.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:52.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:52.469+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:52.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:52.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:52.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:52.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:53.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:53.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:53.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:53.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:53.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:53.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:53.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:53.086+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:53.468+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:53.469+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:53.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:53.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:53.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:53.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:53.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:53.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:53.447+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:54.207 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:54.118+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:54.207 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:54.118+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:54.207 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:54.118+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:54.207 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:54.119+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:54.480 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:54.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:54.480 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:54.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:54.480 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:54.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:54.481 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:54.449+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:54.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:54.479+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:54.479+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:54.479+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:54.480+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:55.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:55.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:55.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:55.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:55.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:55.140+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:55.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:55.141+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:02:55.433+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17444 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:02:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:55.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:55.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:55.451+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:55.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:55.451+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:55.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:55.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:55.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:55.523+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:56.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:56.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:56.099+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:56.100+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:56.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:56.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:56.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:56.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:56.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:56.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:56.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:56.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:56.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:56.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:56.515+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:57.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:57.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:57.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:57.146+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:57.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:57.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:57.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:57.491+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:57.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:57.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:57.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:57.534+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:58.161+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:58.161+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:58.161+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:58.161+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:58.955 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:58.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:58.955 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:58.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:58.475+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:58.475+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:58.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:58.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:58.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:58.956 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:58.578+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:59.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:59.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:59.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:02:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:02:59.187+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:59.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:59.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:59.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:02:59.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:02:59.499+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:02:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:59.531+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:59.531+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:59.531+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:02:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:02:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:02:59.531+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:00.436 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:00.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:00.436 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:00.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:00.436 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:00.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:00.436 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:00.144+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:00.702 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:00.434+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17531 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:00.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:00.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:00.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:00.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:00.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:00.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:00.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:00.487+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:00.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:00.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:00.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:00.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:00.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:00.494+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:01.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:01.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:01.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:01.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:01.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:01.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:01.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:01.142+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:01.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:01.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:01.502+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:01.502+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:01.445+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:01.445+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:01.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:01.445+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:01.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:01.445+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:02.437 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:02.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:02.437 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:02.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:02.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:02.157+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:02.438 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:02.158+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:02.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:02.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:02.473+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:02.474+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:02.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:02.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:02.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:02.437+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:03.589 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:03.387+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:03.589 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:03.387+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:03.589 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:03.387+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:03.589 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:03.387+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:03.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:03.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:03.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:03.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:03.591 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:03.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:03.591 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:03.433+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:03.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:03.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:03.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:03.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:03.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:03.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:03.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:03.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:04.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:04.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:04.418+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:04.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:04.418+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:04.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:04.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:04.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:04.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:04.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:04.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:04.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:04.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:04.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:04.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:04.350+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:05.560 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:05.437+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17617 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:05.560 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:05.434+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:05.560 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:05.434+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:05.561 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:05.434+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:05.561 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:05.435+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:05.561 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:05.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:05.561 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:05.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:05.561 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:05.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:05.562 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:05.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:05.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:05.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:05.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:05.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:05.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:05.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:05.562 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:05.380+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:06.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:06.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:06.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:06.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:06.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:06.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:06.423+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:06.423+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:06.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:06.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:06.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:06.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:06.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:06.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:06.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:06.418+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:07.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:07.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:07.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:07.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:07.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:07.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:07.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:07.183+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:07.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:07.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:07.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:07.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:07.371+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:08.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:08.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:08.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:08.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:08.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:08.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:08.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:08.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:08.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:08.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:08.449 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:08.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:08.449 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:08.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:08.449 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:08.408+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:08.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:08.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:08.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:08.447+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:09.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:09.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:09.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:09.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:09.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:09.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:09.418 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:09.150+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:09.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:09.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:09.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:09.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:09.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:09.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:09.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:09.375+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:09.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:09.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:09.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:09.418+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:10.439 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:10.327+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:10.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:10.327+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:10.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:10.327+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:10.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:10.328+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:10.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:10.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:10.440 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:10.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:10.441 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:10.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:10.441 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:10.137+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:10.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:10.439+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17703 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:10.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:10.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:10.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:10.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:10.449+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:11.286 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:11.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:11.286 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:11.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:11.286 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:11.172+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:11.286 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:11.173+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:11.283+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:11.283+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:11.283+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:11.283+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:11.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:11.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:11.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:11.494+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:12.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:12.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:12.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:12.242+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:12.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:12.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:12.190+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:12.191+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:12.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:12.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:12.454+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:12.454+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:13.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:13.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:13.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:13.200+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:13.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:13.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:13.230+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:13.230+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:13.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:13.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:13.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:13.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:13.497+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:14.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:14.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:14.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:14.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:14.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:14.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:14.524 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:14.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:14.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:14.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:14.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:14.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:14.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:14.231+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:14.525 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:14.232+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:14.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:14.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:14.523+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:14.524+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:15.441 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:15.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:15.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:15.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:15.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:15.229+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:15.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:15.230+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:15.442 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:15.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:15.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:15.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:15.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:15.222+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:15.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:15.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:15.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:15.441+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17788 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:15.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:15.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:15.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:15.563+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:16.526 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:16.270+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:16.526 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:16.270+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:16.270+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:16.271+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:16.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:16.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:16.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:16.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:16.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:16.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:16.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:16.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:16.526+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:17.532 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:17.306+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:17.532 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:17.306+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:17.306+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:17.307+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:17.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:17.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:17.250+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:17.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:17.251+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:17.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:17.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:17.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:17.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:17.532+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:18.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:18.297+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:18.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:18.297+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:18.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:18.297+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:18.513 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:18.298+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:18.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:18.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:18.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:18.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:18.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:18.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:18.514 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:18.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:18.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:18.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:18.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:18.513+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:19.539 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:19.259+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:19.539 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:19.259+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:19.539 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:19.259+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:19.539 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:19.260+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:19.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:19.218+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:19.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:19.218+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:19.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:19.218+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:19.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:19.219+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:19.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:19.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:19.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:19.538+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:20.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:20.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:20.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:20.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:20.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:20.295+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:20.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:20.296+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:20.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:20.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:20.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:20.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:20.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:20.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:20.445 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:20.227+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:20.443+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17875 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:20.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:20.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:20.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:20.544+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:21.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:21.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:21.317+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:21.318+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:21.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:21.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:21.184+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:21.185+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:21.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:21.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:21.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:21.583+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:22.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:22.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:22.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:22.181+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:22.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:22.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:22.359+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:22.360+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:22.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:22.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:22.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:22.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:23.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:23.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:23.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:23.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:23.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:23.387+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:23.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:23.389+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:23.594 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:23.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:23.594 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:23.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:23.594 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:23.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:23.594 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:23.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:23.849 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:23.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:23.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:23.620+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:23.620+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:24.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:24.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:24.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:24.200+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:24.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:24.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:24.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:24.365+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:24.366+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:24.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:24.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:24.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:24.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:24.622+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:25.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:25.445+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 17962 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:25.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:25.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:25.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:25.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:25.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:25.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:25.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:25.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:25.399+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:25.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:25.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:25.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:25.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:25.599+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:26.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:26.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:26.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:26.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:26.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:26.425+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:26.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:26.426+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:26.556 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:26.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:26.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:26.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:26.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:26.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:26.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:26.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:26.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:26.555+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:26.555+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:26.555+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:26.556+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:27.559 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:27.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:27.559 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:27.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:27.559 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:27.459+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:27.559 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:27.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:27.560 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:27.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:27.560 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:27.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:27.560 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:27.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:27.560 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:27.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:27.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:27.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:27.558+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:27.559+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:28.563 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:28.414+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:28.415+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:28.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:28.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:28.564 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:28.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:28.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:28.235+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:28.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:28.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:28.563+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:28.563+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:29.519 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:29.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:29.519 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:29.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:29.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:29.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:29.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:29.375+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:29.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:29.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:29.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:29.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:29.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:29.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:29.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:29.241+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:29.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:29.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:29.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:29.519+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:30.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:30.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:30.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:30.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:30.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:30.413+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:30.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:30.413+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:30.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:30.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:30.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:30.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:30.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:30.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:30.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:30.213+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:30.447+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18049 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:30.474+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:30.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:30.474+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:30.474+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:30.474+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:31.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:31.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:31.399+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:31.400+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:31.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:31.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:31.428+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:31.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:31.179+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:31.179+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:31.179+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:31.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:31.179+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:32.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:32.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:32.478 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:32.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:32.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:32.226+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:32.361+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:32.361+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:32.361+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:32.479 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:32.361+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:32.477+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:32.477+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:32.477+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:32.478+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:33.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:33.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:33.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:33.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:33.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:33.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:33.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:33.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:33.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:33.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:33.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:33.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:33.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:33.385+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:33.431+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:33.431+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:33.431+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:33.593 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:33.431+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:34.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:34.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:34.423+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:34.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:34.424+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:34.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:34.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:34.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:34.240+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:34.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:34.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:34.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:34.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:34.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:35.574 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:35.449+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18136 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:35.574 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:35.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:35.574 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:35.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:35.574 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:35.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:35.574 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:35.449+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:35.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:35.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:35.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:35.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:35.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:35.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:35.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:35.238+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:35.575 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:35.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:35.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:35.406+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:35.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:35.407+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:36.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:36.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:36.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:36.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:36.436+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:36.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:36.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:36.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:36.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:36.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:36.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:36.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:36.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:36.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:36.473+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:37.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:37.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:37.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:37.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:37.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:37.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:37.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:37.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:37.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:37.604 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:37.440+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:37.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:37.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:37.307+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:37.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:37.308+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:38.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:38.266+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:38.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:38.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:38.446+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:38.447+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:38.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:38.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:38.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:38.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:38.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:38.470+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:38.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:38.471+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:39.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:39.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:39.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:39.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:39.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:39.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:39.406+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:39.406+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:39.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:39.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:39.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:39.447+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:39.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:39.448+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:40.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:40.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:40.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:40.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:40.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:40.408+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:40.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:40.409+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:40.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:40.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:40.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:40.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:40.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:40.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:40.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:40.268+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:40.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:40.451+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18224 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:40.418+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:40.418+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:40.418+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:40.418+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:41.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:41.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:41.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:41.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:41.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:41.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:41.378+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:41.378+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:41.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:41.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:41.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:41.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:41.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:41.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:41.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:41.249+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:42.450+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:42.450+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:42.450+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:42.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:42.450+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:42.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:42.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:42.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:42.352+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:42.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:42.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:42.217+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:42.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:42.218+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:43.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:43.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:43.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:43.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:43.404+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:43.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:43.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:43.384+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:43.385+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:43.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:43.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:43.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:43.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:43.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:43.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:43.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:43.225+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:44.403+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:44.403+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:44.403+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:44.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:44.403+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:44.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:44.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:44.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:44.388+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:44.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:44.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:44.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:44.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:44.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:45.577 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:45.453+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18311 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:45.577 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:45.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:45.577 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:45.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:45.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:45.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:45.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:45.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:45.428+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:45.578 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:45.429+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:45.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:45.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:45.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:45.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:45.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:45.425+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:45.579 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:45.425+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:46.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:46.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:46.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:46.308+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:46.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:46.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:46.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:46.422+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:46.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:46.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:46.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:46.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:46.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:46.430+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:46.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:46.431+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:47.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:47.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:47.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:47.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:47.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:47.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:47.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:47.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:47.440+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:47.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:47.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:47.412+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:47.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:47.412+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:48.380+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:48.380+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:48.380+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:48.381+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:48.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:48.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:48.456+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:48.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:48.457+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:48.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:48.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:48.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:48.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:48.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:48.367+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:48.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:48.368+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:49.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:49.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:49.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:49.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:49.459+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:49.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:49.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:49.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:49.409+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:49.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:49.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:49.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:49.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:49.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:49.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:12.360298+0000 (oldest deadline 2024-03-24T10:53:32.360298+0000) 2024-03-24T11:03:49.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:49.391+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:50.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:50.455+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18398 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:50.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:50.424+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:50.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:50.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:50.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:50.437+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:50.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:50.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:50.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:50.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:50.443+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:21.043471+0000 (oldest deadline 2024-03-24T10:53:41.043471+0000) 2024-03-24T11:03:50.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:50.443+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:51.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:51.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:51.462+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:51.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:51.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:51.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:51.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:52.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:52.483+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:52.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:52.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:52.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:52.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:53.486+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:53.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:53.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:53.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:53.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:53.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:53.438+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:54.453+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:54.478+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:54.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:54.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:54.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:54.432+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:03:55.458+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18485 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:03:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:55.436+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:55.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:55.464+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:55.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:55.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:55.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:55.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:55.452+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:56.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:56.465+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:56.485+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:56.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:56.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:56.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:56.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:56.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:53:22.215172+0000 (oldest deadline 2024-03-24T10:53:42.215172+0000) 2024-03-24T11:03:56.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:56.491+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:57.484+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:57.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:57.515+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:57.511+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:58.516+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:58.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:58.521+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:58.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:58.519+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:59.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:03:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:03:59.530+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:59.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:03:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:03:59.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:03:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:03:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:03:59.513+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:00.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:00.534+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:00.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:00.476+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:00.525+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:00.952 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:00.460+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18573 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:01.563+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:01.450+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:01.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:02.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:02.549+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:02.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:02.583+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:02.953 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:02.467+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:03.579+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:03.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:03.512+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:03.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:03.547+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:04.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:04.585+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:04.533+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:04.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:04.508+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:05.462+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18660 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:05.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:05.461+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:05.505+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:05.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:05.544+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:06.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:06.469+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:06.524+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:06.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:07.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:07.500+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:07.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:07.553+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:07.405+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:08.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:08.550+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:08.425+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:08.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:09.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:09.398+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:09.561+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:09.407+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:10.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:10.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:10.368+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:10.950 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:10.463+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18747 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:10.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:10.542+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:11.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:11.505+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:11.446+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:11.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:11.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:12.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:12.423+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:12.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:12.501+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:12.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:13.408+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:13.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:13.526+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:13.420+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:14.409+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:14.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:14.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:14.519+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:14.520+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:14.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:14.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:14.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:14.468+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:15.700 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:15.466+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18834 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:15.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:15.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:15.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:15.435+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:15.435+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:15.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:15.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:15.487+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:15.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:15.487+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:16.410+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:16.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:16.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:16.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:16.428+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:16.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:16.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:16.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:16.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:16.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:16.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:16.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:16.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:17.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:17.437+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:17.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:17.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:17.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:17.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:17.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:17.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:17.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:17.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:17.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:17.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:17.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:17.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:17.705 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:17.413+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:17.705 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:17.414+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:18.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:18.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:18.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:18.388+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:18.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:18.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:18.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:18.490+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:18.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:18.491+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:18.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:18.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:18.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:18.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:18.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:18.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:18.463+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:19.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:19.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:19.429+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:19.430+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:19.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:19.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:19.532+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:19.533+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:19.403+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:19.403+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:19.403+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:19.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:19.404+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:20.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:20.468+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 18922 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:20.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:20.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:20.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:20.548+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:20.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:20.549+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:20.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:20.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:20.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:20.440+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:20.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:20.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:20.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:20.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:20.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:20.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:20.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:20.448+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:21.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:21.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:21.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:21.598+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:21.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:21.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:21.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:21.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:21.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:21.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:21.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:21.489+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:21.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:21.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:21.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:21.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:21.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:21.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:21.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:22.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:22.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:22.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:22.476+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:22.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:22.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:22.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:22.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:22.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:22.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:22.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:22.486+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:23.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:23.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:23.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:23.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:23.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:23.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:23.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:23.442+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:23.621+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:23.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:23.621+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:23.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:23.621+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:23.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:23.621+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:24.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:24.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:24.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:24.406+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:24.430+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:24.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:24.430+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:24.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:24.430+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:24.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:24.431+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:24.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:24.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:24.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:24.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:24.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:24.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:24.704 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:24.606+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:25.686 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:25.470+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19009 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:25.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:25.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:25.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:25.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:25.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:25.391+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:25.687 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:25.391+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:25.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:25.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:25.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:25.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:25.589+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:25.688 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:25.589+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:25.689 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:25.589+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:25.689 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:25.590+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:26.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:26.585+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:26.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:26.585+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:26.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:26.585+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:26.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:26.586+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:26.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:26.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:26.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:26.357+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:26.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:26.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:26.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:26.705 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:26.353+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:26.705 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:26.354+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:27.625 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:27.625 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:27.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:27.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:27.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:27.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:27.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:27.626 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:27.344+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:27.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:27.625+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:27.625+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:27.625+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:27.626+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:28.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:28.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:28.346+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:28.347+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:28.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:28.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:28.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:28.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:28.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:28.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:28.376+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:28.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:28.663+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:28.663+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:28.663+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:28.664+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:29.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:29.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:29.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:29.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:29.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:29.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:29.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:29.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:29.676 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:29.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:29.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:29.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:29.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:29.333+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:29.677 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:29.334+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:29.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:29.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:29.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:29.676+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:30.420 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:30.363+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:30.420 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:30.363+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:30.421 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:30.363+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:30.421 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:30.364+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:30.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:30.472+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19098 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:30.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:30.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:30.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:30.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:30.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:30.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:30.626+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:30.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:30.627+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:31.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:31.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:31.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:31.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:31.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:31.343+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:31.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:31.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:31.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:31.624 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:31.426+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:31.623+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:31.623+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:31.623+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:31.623+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:32.642 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:32.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:32.642 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:32.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:32.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:32.380+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:32.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:32.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:32.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:32.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:32.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:32.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:32.642+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:32.642+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:32.642+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:32.643+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:33.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:33.620+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:33.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:33.620+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:33.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:33.620+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:33.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:33.620+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:33.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:33.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:33.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:33.435+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:33.405+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:33.405+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:33.405+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:33.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:33.405+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:34.384+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:34.384+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:34.384+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:34.384+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:34.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:34.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:34.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:34.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:34.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:34.614+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:34.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:34.615+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:34.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:34.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:34.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:34.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:34.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:34.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:34.425+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:35.474+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19187 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:35.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:35.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:35.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:35.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:35.385+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:35.461+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:35.461+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:35.461+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:35.462+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:35.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:35.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:35.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:35.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:35.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:35.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:35.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:35.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:36.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:36.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:36.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:36.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:36.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:36.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:36.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:36.506+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:36.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:36.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:36.600 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:36.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:36.601 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:36.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:36.601 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:36.348+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:36.600+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:37.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:37.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:37.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:37.546+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:37.547+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:37.332+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:37.332+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:37.636 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:37.332+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:37.637 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:37.332+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:37.635+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:37.635+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:37.635+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:37.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:37.636+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:38.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:38.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:38.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:38.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:38.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:38.584+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:38.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:38.585+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:38.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:38.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:38.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:38.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:38.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:38.376+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:38.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:38.377+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:38.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:38.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:38.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:38.680+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:39.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:39.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:39.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:39.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:39.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:39.573+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:39.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:39.574+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:39.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:39.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:39.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:39.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:39.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:39.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:39.681 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:39.374+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:39.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:39.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:39.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:39.680+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:40.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:40.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:40.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:40.399+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:40.808 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:40.476+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19277 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:40.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:40.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:40.704+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:40.705+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:40.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:40.809 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:40.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:40.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:40.615+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:40.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:40.615+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:41.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:41.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:41.636+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:41.637+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:41.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:41.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:41.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:41.404+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:41.741+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:41.741+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:41.741+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:41.742+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:42.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:42.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:42.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:42.397+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:42.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:42.397+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:42.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:42.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:42.663+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:42.664+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:42.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:42.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:42.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:42.735+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:42.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:42.735+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:43.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:43.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:43.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:43.429+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:43.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:43.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:43.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:43.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:43.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:43.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:43.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:43.699+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:43.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:43.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:43.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:43.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:43.770+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:44.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:44.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:44.449+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:44.449+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:44.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:44.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:44.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:44.798+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:45.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:44.715+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:45.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:44.715+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:45.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:44.715+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:45.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:44.715+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:45.761 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:45.479+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19366 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:45.762 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:45.471+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:45.762 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:45.471+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:45.762 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:45.471+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:45.762 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:45.471+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:45.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:45.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:45.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:45.847+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:45.761+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:45.761+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:45.761+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:45.762+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:46.897 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:46.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.897 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:46.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.897 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:46.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:46.897 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:46.725+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:46.898 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:46.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:46.898 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:46.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:46.898 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:46.486+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:46.898 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:46.486+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:46.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:46.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:46.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:46.897+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:47.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:47.680+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:47.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:47.680+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:47.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:47.680+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:47.925 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:47.681+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:47.925 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:47.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:47.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:47.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:47.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:47.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:47.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:47.497+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:48.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:47.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:47.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:47.924+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:47.925+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:48.906 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:48.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:48.906 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:48.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:48.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:48.638+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:48.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:48.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:48.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:48.907 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:48.505+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:49.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:48.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:48.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:48.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:48.906+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:49.861 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:49.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:49.861 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:49.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:49.862 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:49.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:49.862 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:49.502+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:49.862 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:49.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:49.862 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:49.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:49.862 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:49.644+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:49.863 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:49.645+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:50.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:49.861+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:50.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:49.861+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:50.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:49.861+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:50.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:49.861+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:50.890 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:50.481+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19455 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:50.891 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:50.660+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:50.891 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:50.660+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:50.891 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:50.660+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:50.891 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:50.661+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:50.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:50.459+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:50.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:50.459+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:50.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:50.459+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:50.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:50.459+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:50.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:50.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:50.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:50.907+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:51.883 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:51.694+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:51.883 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:51.694+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:51.883 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:51.694+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:51.883 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:51.694+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:51.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:51.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:51.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:51.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:51.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:51.456+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:51.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:51.456+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:52.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:51.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:51.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:51.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:51.883+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:52.698 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:52.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:52.698 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:52.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:52.699 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:52.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:52.699 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:52.407+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:52.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:52.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:52.698+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:52.699+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:52.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:52.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:52.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:52.920+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:53.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:53.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:53.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:53.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:53.439+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:53.898+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:53.898+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:53.898+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:53.899+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:53.737+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:53.737+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:53.737+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:53.738+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:54.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:54.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:54.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:54.421+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:54.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:54.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:54.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:54.888+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:54.713+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:54.713+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:54.713+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:54.714+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:55.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:04:55.483+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19544 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:04:55.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:55.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:55.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:55.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:55.405+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:55.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:55.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:55.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:55.872+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:55.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:55.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:55.752+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:56.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:55.753+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:56.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:56.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:56.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:56.439+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:56.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:56.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:56.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:56.830+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:56.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:56.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:56.787+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:56.788+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:57.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:57.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:57.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:57.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:57.430+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:57.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:57.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:57.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:57.873+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:57.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:57.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:57.812+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:57.813+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:58.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:58.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:58.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:58.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:58.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:58.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:58.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:58.453+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:58.815+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:58.815+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:58.815+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:04:59.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:58.816+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:58.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:58.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:58.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:04:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:58.831+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:04:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:59.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:59.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:59.437+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:04:59.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:04:59.437+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:00.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:59.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:00.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:59.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:00.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:59.851+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:00.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:04:59.852+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:00.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:00.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:00.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:00.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:04:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:04:59.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:00.867 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:00.486+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19633 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:00.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:00.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:00.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:00.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:00.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:00.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:00.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:00.475+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:00.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:00.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:00.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:00.877+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:01.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:00.867+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:00.867+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:00.867+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:01.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:00.868+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:01.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:01.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:01.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:01.438+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:02.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:01.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:01.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:01.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:01.923+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:02.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:01.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:02.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:01.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:01.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:01.901+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:02.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:02.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:02.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:02.429+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:02.430+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:02.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:02.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:02.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:02.963+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:03.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:02.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:03.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:02.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:03.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:02.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:03.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:02.937+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:03.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:03.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:03.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:03.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:03.479+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:03.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:03.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:03.926+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:04.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:03.927+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:04.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:04.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:04.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.007+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:04.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:04.489+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:04.489+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:04.489+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:04.490+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:04.968+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:04.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:04.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:04.951+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:04.952+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:05.937 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:05.488+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19722 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:05.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:05.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:05.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:05.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:05.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:05.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:05.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:05.476+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:05.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:05.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:05.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:05.938+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:06.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:05.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:06.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:05.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:06.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:05.981+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:06.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:05.982+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:06.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:06.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:06.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:06.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:06.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:06.453+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:06.896 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:06.453+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:06.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:06.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:06.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:06.895+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:07.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:07.021+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:07.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:07.021+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:07.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:07.021+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:07.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:07.022+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:07.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:07.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:07.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:07.412+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:07.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:07.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:07.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:07.884+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:08.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:08.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:08.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:08.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:08.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:08.046+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:08.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:08.047+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:08.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:08.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:08.372+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:08.372+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:08.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:08.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:08.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:08.872+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:09.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:09.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:09.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:09.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:09.064+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:09.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:09.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:09.374+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:09.375+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:10.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:10.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:10.172 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:10.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:10.107+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:10.108+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:09.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:09.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:09.892+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:10.173 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:09.893+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:10.365+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:10.365+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:10.365+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:10.365+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:10.829 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:10.490+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19811 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:11.093 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:11.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:11.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:11.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:10.916+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:11.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:11.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:11.323+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:11.323+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:11.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:11.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:11.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:11.093+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:11.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:11.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:11.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:11.870+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:12.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:12.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:12.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:12.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:12.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:12.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:12.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:12.050+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:12.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:12.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:12.310+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:12.310+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:12.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:12.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:12.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:12.849+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:13.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:13.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:13.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:13.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:13.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:13.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:13.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:13.074+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:13.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:13.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:13.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:13.325+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:14.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:13.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:14.096 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:13.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:14.097 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:13.818+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:14.097 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:13.819+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:14.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:14.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:14.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:14.096+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:14.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:14.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:14.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:14.343+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:14.343+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:15.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:14.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:15.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:14.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:15.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:14.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:15.070 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:14.770+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:15.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:15.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:15.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:15.069+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:15.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:15.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:15.309+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:15.310+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:15.749 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:15.492+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19900 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:16.091 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:15.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:16.092 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:15.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:16.092 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:15.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:16.092 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:15.749+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:16.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:16.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:16.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:16.095+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:16.263+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:16.264+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:16.264+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:16.264+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:17.065 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:16.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:17.066 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:16.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:17.066 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:16.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:17.066 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:16.771+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:17.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:17.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:17.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:17.065+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:17.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:17.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:17.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:17.219+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:18.072 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:17.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:18.072 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:17.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:18.072 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:17.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:18.073 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:17.773+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:18.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:18.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:18.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:18.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:18.192+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:18.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:18.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:18.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:18.071+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:19.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:18.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:19.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:18.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:19.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:18.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:19.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:18.783+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:19.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:19.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:19.100+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:19.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:19.101+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:19.231+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:19.231+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:20.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:20.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:20.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:20.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:20.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:20.174 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:19.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:20.174 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:19.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:20.174 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:19.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:20.174 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:19.807+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:20.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:20.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:20.234+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:20.235+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:20.844 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:20.494+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 19990 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:21.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:20.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:20.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:20.844+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:20.844+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:21.022+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:21.022+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:21.131 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:21.022+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:21.132 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:21.022+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:21.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:21.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:21.240+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:21.240+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:22.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:22.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:22.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:22.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:22.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:21.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:21.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:21.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:21.841+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:22.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:22.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:22.202+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:22.202+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:23.117 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:22.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:23.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:22.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:23.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:22.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:23.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:22.861+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:23.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:23.021+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:23.119 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:23.022+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:23.119 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:23.022+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:23.119 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:23.022+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:23.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:23.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:23.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:23.218+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:23.218+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:24.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:24.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:24.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:24.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:23.868+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:23.868+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:23.868+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:23.869+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:24.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:24.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:24.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:24.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:24.217+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:25.178 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:25.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:25.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:25.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:25.065+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:24.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:24.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:24.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:25.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:24.861+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:25.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:25.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:25.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:25.178+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:25.874 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:25.497+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20079 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:26.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:26.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:26.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:26.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:26.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:26.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:26.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:26.075+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:26.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:25.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:26.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:25.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:26.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:25.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:26.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:25.874+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:26.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:26.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:26.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:26.221+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:26.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:26.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:26.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:26.886+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:27.029+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:27.029+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:27.029+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:27.030+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:27.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:27.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:27.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:27.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:27.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:27.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:27.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:27.886+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:28.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:28.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:28.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:28.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:28.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:28.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:28.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:28.040+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:28.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:28.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:28.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:28.213+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:29.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:29.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:29.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:29.088+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:28.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:28.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:28.888+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:28.889+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:29.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:29.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:29.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:29.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:30.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:29.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:30.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:29.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:30.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:29.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:30.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:29.840+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:30.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:30.122+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:30.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:30.122+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:30.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:30.122+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:30.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:30.123+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:30.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:30.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:30.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:30.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:30.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:30.280+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:30.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:30.281+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:30.841 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:30.499+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20168 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:31.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:30.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:31.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:30.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:31.121 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:30.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:31.122 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:30.841+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:31.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:31.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:31.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:31.121+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:31.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:31.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:31.233+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:31.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:31.234+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:32.084 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:31.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:32.084 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:31.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:32.084 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:31.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:32.085 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:31.834+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:32.085 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:32.083+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:32.085 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:32.083+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:32.085 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:32.083+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:32.084+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:32.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:32.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:32.279+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:32.280+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:33.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:32.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:33.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:32.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:33.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:32.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:33.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:32.851+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:33.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:33.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:33.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:33.133+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:33.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:33.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:33.293+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:33.294+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:34.085 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:33.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:34.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:33.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:34.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:33.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:34.086 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:33.829+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:34.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:34.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:34.085+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:34.086+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:34.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:34.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:34.313+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:34.314+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:35.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:34.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:35.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:34.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:35.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:34.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:35.116 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:34.846+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:35.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:35.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:35.115+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:35.116+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:35.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:35.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:35.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:35.303+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:35.819 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:35.502+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20257 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:36.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:36.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:36.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:36.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:35.819+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:36.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:36.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:36.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:36.103+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:36.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:36.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:36.327+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:36.327+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:37.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:36.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:37.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:36.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:37.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:36.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:37.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:36.780+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:37.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:37.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:37.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:37.142+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:37.370+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:38.138 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:37.826+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:38.138 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:37.827+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:38.138 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:37.827+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:38.138 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:37.827+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:38.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:38.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:38.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:38.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:38.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:38.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:38.389 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:38.138+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:38.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:38.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:38.388+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:38.389+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:39.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:38.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:39.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:38.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:39.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:38.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:39.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:38.856+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:39.407 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:39.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:39.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:39.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:39.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:39.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:39.408 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:39.129+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:39.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:39.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:39.407+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:39.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:39.407+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:40.123 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:39.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:40.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:39.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:40.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:39.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:40.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:39.809+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:40.390 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:40.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:40.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:40.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:40.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:40.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:40.391 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:40.124+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:40.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:40.504+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20346 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:40.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:40.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:40.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:40.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:41.170 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:40.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:41.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:40.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:41.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:40.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:41.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:40.834+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:41.427 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:41.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:41.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:41.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:41.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:41.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:41.428 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:41.171+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:41.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:41.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:41.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:41.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:41.427+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:42.156 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:41.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:42.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:41.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:42.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:41.845+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:42.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:41.846+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:42.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:42.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:42.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:42.385+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:42.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:42.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:42.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:42.156+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:42.156+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:43.126 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:42.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:43.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:42.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:43.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:42.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:43.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:42.807+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:43.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:43.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:43.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:43.127+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:43.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:43.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:43.366+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:43.367+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:44.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:44.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:44.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:44.094+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:43.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:43.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:43.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:43.847+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:44.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:44.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:44.339+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:44.339+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:45.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:44.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:45.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:44.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:45.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:44.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:45.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:44.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:45.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:45.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:45.350+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:45.350+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:45.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:45.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:45.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:45.135+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:45.825 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:45.506+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20434 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:46.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:45.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:46.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:45.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:46.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:45.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:46.103 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:45.826+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:46.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:46.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:46.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:46.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:46.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:46.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:46.380 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:46.103+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:46.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:46.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:46.379+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:46.380+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:47.064 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:46.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:47.064 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:46.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:47.064 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:46.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:47.064 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:46.805+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:47.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:47.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:47.419 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:47.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:47.420 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:47.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:47.420 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:47.064+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:47.419+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:47.419+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:47.419+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:47.419+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:47.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:47.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:47.840+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:47.841+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:48.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:48.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:48.077+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:48.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:48.439+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:48.439+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:48.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:48.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:48.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:48.819+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:49.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:49.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:49.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:49.050+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:49.810 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:49.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:49.810 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:49.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:49.810 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:49.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:49.811 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:49.461+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:50.175 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:49.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:50.175 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:49.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:50.175 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:49.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:50.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:49.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:50.176 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.006+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:50.176 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.006+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:50.176 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.006+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:50.176 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.007+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:50.765 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:50.508+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20519 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:50.766 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:50.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:50.766 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:50.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:50.766 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:50.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:50.766 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:50.487+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:50.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:50.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:50.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:50.830+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:51.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:50.981+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:51.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:51.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:51.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:51.519+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:51.997+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:51.997+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:52.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:51.997+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:52.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:51.997+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:51.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:51.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:51.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:51.843+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:52.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:52.530+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:52.884 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:52.531+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:52.885 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:52.531+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:52.885 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:52.531+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.017+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:52.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:52.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:52.884+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:52.885+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:53.508+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:53.508+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:53.509+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:53.509+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:53.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:53.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:53.877+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:53.878+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.979+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:53.980+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:54.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:54.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:54.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:54.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:54.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:54.505+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:54.913 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:54.506+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:54.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:54.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:54.911+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:54.912+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:55.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:54.973+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:54.973+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:54.973+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:55.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:54.973+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:55.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:05:55.511+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20599 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:05:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:55.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:55.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:55.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:55.550+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:56.413 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:55.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:55.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:55.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:55.958+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:55.968+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:55.968+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:56.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:55.968+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:56.415 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:55.969+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:56.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:56.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:56.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:56.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:56.567+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:56.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:56.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:56.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:56.970+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.017+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:57.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:57.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:57.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:57.602+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:57.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:57.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:57.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:58.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:57.969+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:57.988+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:58.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:58.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:58.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:58.567+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:59.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:59.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:59.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:05:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:05:59.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:59.026+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:59.026+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:59.026+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:05:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:05:59.027+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:05:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:59.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:59.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:59.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:05:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:05:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:05:59.534+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:00.000+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:00.000+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:00.000+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:00.000+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:00.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:00.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:00.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:00.053+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:00.513+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20680 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:00.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:00.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:00.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:00.573+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:01.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:01.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:01.045+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:01.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:01.046+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:01.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:01.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:01.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:01.096+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:01.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:01.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:01.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:01.540+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:02.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:02.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:02.038+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:02.038+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:02.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:02.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:02.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:02.122+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:02.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:02.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:02.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:02.525+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:03.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:03.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:03.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:03.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:03.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:03.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:03.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:03.079+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:03.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:03.168+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:03.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:03.168+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:03.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:03.168+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:03.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:03.168+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:03.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:03.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:03.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:03.512+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:04.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:04.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:04.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:04.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:04.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:04.066+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:04.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:04.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:04.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:04.200+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:04.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:04.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:04.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:04.555+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:05.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:05.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:05.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:05.198+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:05.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:05.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:05.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:05.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:05.515+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20759 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:05.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:05.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:05.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:05.519+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:06.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:06.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:06.019+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:06.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:06.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:06.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:06.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:06.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:06.226+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:06.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:06.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:06.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:06.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:06.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:06.484+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:07.260 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:07.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:07.260 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:07.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:07.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:07.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:07.261 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:07.008+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:07.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:07.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:07.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:07.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:07.464+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:07.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:07.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:07.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:07.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:08.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:08.031+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:08.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:08.031+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:08.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:08.031+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:08.284 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:08.032+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:08.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:08.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:08.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:08.491+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:08.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:08.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:08.283+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:08.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:09.328 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:09.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:09.328 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:09.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:09.328 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:09.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:09.328 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:09.022+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:09.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:09.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:09.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:09.328+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:09.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:09.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:09.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:09.475+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:10.332 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:10.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:10.332 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:10.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:10.332 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:10.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:10.333 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:10.026+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:10.642 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:10.518+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20840 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:10.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:10.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:10.333+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:10.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:10.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:10.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:10.643 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:10.495+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:11.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:11.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:11.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:11.295 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.004+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:11.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:11.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:11.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:11.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:11.516+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:12.269 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:12.270 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:12.270 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:12.270 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:11.995+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:12.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:12.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:12.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:12.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:12.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:12.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:12.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:12.316+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:12.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:12.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:12.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:12.520+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:13.325 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:13.325 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:13.325 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:13.326 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:13.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:13.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:13.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:13.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:13.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:13.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:13.325+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:13.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:13.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:14.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:14.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:14.495+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:14.496+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:14.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:14.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:14.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:14.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:15.275 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:14.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:15.275 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:14.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:15.276 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:14.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:15.276 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:14.998+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:15.531 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:15.520+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 20925 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:15.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:15.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:15.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:15.518+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:15.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:15.532 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:15.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:15.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:15.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:15.533 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:15.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:16.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:15.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:16.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:15.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:16.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:15.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:16.256 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:15.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:16.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:16.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:16.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:16.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:16.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:16.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:16.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:16.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:16.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:16.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:16.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:16.568+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:17.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:16.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:17.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:16.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:17.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:16.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:17.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:16.959+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:17.579 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:17.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:17.579 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:17.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:17.579 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:17.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:17.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:17.238+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:17.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:17.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:17.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:17.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:17.579+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:18.244 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:17.952+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:18.244 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:17.952+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:18.244 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:17.952+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:18.244 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:17.953+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:18.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:18.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:18.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:18.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:18.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:18.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:18.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:18.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:18.950 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:18.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:18.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:18.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:18.535+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:19.245 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:18.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:19.245 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:18.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:19.245 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:18.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:19.245 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:18.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:19.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:19.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:19.550 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:19.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:19.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:19.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:19.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:19.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:19.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:19.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:19.550+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:19.550+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:20.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:20.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:20.023+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:20.023+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:20.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:20.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:20.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:20.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:20.522+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21016 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:20.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:20.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:20.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:20.585+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:21.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:21.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:21.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:21.055+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:21.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:21.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:21.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:21.225+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:21.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:21.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:21.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:21.600+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:22.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:22.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:22.199+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:22.199+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:22.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:22.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:22.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:22.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:22.070+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:22.615+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:22.615+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:22.615+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:22.616+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:23.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:23.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:23.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:23.034+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:23.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:23.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:23.183+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:23.184+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:23.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:23.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:23.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:23.634+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.002+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:24.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.002+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:24.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:24.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:24.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:24.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:24.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:24.203+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:24.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:24.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:24.640+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:24.641+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:25.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:25.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:25.165+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:25.166+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:24.958+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:25.524+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21104 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:25.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:25.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:25.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:25.657+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:25.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:25.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:25.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:25.993+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:26.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:26.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:26.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:26.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:26.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:26.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:26.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:26.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:26.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:26.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:26.948 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:26.669+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:26.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:26.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:26.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:26.947+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:27.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:27.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:27.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:27.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:27.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:27.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:27.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:27.228+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:27.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:27.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:27.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:27.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:27.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:27.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:27.944 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:27.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:27.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:27.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:27.941+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:27.941+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:28.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:28.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:28.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:28.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:28.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:28.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:28.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:28.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:28.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:28.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:28.629+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:28.629+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:29.236 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:28.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:29.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:28.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:29.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:28.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:29.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:28.965+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:29.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:29.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:29.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:29.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:29.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:29.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:29.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:29.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:29.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:29.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:29.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:29.673+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:30.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:30.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:30.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:30.196+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:29.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:29.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:29.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:29.985+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:30.950 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:30.526+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21193 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:30.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:30.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:30.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:30.664+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:30.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:31.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:30.956+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:31.163+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:31.163+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:31.163+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:31.164+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:31.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:31.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:31.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:31.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:31.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:31.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:31.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:31.653+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:31.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:31.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:31.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:31.945+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:32.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:32.155+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:32.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:32.155+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:32.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:32.155+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:32.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:32.156+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:32.908 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:32.637+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:32.908 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:32.637+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:32.908 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:32.637+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:32.908 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:32.637+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:33.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:32.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:32.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:32.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:32.908+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:33.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:33.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:33.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:33.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:33.172+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:33.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:33.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:33.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:33.873+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:33.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:33.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:33.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:33.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:33.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:33.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:33.644+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:34.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:34.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:34.137+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:34.138+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:34.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:34.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:34.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:34.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:34.829+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:34.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:34.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:34.619+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:34.619+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:35.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:35.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:35.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:35.154+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:35.820 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:35.529+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21281 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:35.820 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:35.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:35.822 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:35.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:35.822 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:35.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:35.822 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:35.663+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:36.181 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.181 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:35.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:36.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:36.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:36.180+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:36.181+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:36.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:36.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:36.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:36.655+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:36.839+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:36.839+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:36.839+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:36.840+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:37.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:37.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:37.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:37.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:37.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:37.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:37.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:37.224+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:37.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:37.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:37.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:37.653+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:37.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:37.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:37.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:37.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:37.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:37.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:37.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:37.850+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:38.631 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:38.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:38.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:38.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:38.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:38.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:38.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:38.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:38.891 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:38.631+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:38.891 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:38.631+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:38.891 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:38.631+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:38.891 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:38.631+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:38.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:38.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:38.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:38.891+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:39.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:39.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:39.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:39.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:39.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:39.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:39.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:39.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:39.909+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:39.909+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:39.909+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:39.910+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:39.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:39.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:39.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:39.679+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:40.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:40.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:40.253+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:40.254+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:40.865 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:40.532+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21372 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:40.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:40.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:40.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:40.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:40.865 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:40.711+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:40.866 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:40.712+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:40.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:40.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:40.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:40.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:41.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:41.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:41.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:41.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:41.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:41.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:41.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:41.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:41.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:41.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:41.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:41.863+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:41.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:41.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:41.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:41.689+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:42.696 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:42.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:42.696 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:42.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:42.696 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:42.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:42.697 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:42.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:42.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:42.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:42.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:42.864+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:42.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:42.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:42.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:42.696+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:43.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:43.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:43.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:43.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:43.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:43.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:43.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:43.274+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:43.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:43.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:43.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:43.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:43.668+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:43.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:43.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:43.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:43.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:43.822+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:44.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:44.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:44.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:44.230+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:44.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:44.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:44.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:44.707+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:44.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:44.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:44.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:44.825+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:45.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:45.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:45.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:45.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:45.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:45.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:45.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:45.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:45.868 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:45.534+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21460 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:45.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:45.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:45.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.868 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:45.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:45.869 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:45.723+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:45.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:45.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:45.867+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:45.868+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:46.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:46.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:46.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:46.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:46.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:46.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:46.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:46.719+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:46.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:46.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:46.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:46.838+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:47.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:47.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:47.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:47.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:47.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:47.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:47.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:47.826+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:47.725+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:47.725+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:47.725+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:47.725+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:48.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:48.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:48.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:48.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:48.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:48.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:48.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:48.793+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:48.701+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:48.701+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:48.701+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:48.701+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:49.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:49.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:49.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:49.211+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:50.170 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:49.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.171 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:49.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.171 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:49.748+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.171 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:49.748+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:50.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:49.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:49.777+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:50.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:50.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:50.170+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:50.171+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:50.537+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21550 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:50.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:50.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:50.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:50.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:50.772+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:50.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:50.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:50.776+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:50.777+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:51.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:51.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:51.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:51.197+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:52.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:51.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:52.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:51.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:52.176 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:51.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:52.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:51.775+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:52.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:51.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:52.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:51.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:52.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:51.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:52.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:51.808+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:52.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:52.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:52.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:52.177+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:53.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:52.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:53.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:52.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:53.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:52.751+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:53.139 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:52.752+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:53.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:52.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:53.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:52.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:53.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:52.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:53.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:52.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:53.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:53.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:53.138+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:53.139+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:54.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:53.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:54.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:53.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:54.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:53.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:54.110 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:53.774+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:54.110 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:53.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:54.110 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:53.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:54.110 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:53.791+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:54.110 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:53.791+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:54.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:54.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:54.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:54.109+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:55.112 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:54.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:55.112 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:54.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:55.112 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:54.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:55.112 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:54.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:55.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:54.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:55.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:54.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:55.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:54.781+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:55.113 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:54.781+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:55.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:55.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:55.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:55.112+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:55.804 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:06:55.539+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21638 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:06:56.147 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:55.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:56.147 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:55.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:56.147 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:55.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:56.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:55.805+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:56.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:55.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:56.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:55.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:56.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:55.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:56.148 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:55.823+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:56.146+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:56.146+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:56.146+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:56.147+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:57.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:56.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:56.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:56.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:56.853+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:56.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:56.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:56.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:57.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:56.857+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:57.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:57.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:57.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:57.137+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:58.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:57.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:58.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:57.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:58.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:57.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:58.127 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:57.885+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:58.127 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:57.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:58.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:57.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:58.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:57.870+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:58.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:57.871+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:58.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:58.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:58.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:58.127+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:59.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:58.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:59.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:58.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:58.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:58.843+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:58.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:58.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:58.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:06:59.130 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:58.887+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:06:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:59.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:59.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:59.129+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:06:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:06:59.130+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:00.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:00.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:59.864+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:06:59.865+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:59.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:59.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:59.887+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:00.135 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:06:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:06:59.887+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:00.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:00.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:00.133+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:00.134+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:00.863 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:00.542+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21729 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:01.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:00.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:01.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:00.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:01.144 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:00.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:01.145 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:00.882+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:01.145 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:00.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:01.145 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:00.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:01.145 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:00.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:01.145 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:00.863+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:01.421 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:01.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:01.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:01.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:01.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:01.144+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:01.422 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:01.144+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:02.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:01.856+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:02.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:01.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:02.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:01.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:02.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:01.857+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:02.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:01.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:02.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:01.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:02.138 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:01.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:02.138 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:01.841+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:02.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:02.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:02.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:02.137+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:03.176 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:02.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:03.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:02.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:03.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:02.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:03.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:02.858+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:03.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:02.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:03.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:02.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:03.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:02.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:03.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:02.869+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:03.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:03.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:03.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:03.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:03.177+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:04.177 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:03.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:04.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:03.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:04.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:03.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:04.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:03.874+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:03.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:04.178 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:03.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:04.179 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:03.862+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:04.179 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:03.862+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:04.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:04.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:04.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:04.177+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:04.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:04.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:04.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:05.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:04.922+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:04.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:04.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:04.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:04.874+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:05.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:05.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:05.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:05.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:05.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:05.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:05.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:05.203+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:05.858 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:05.544+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21817 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:05.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:05.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:05.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:06.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:05.970+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:05.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:05.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:05.858+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:06.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:05.858+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:06.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:06.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:06.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:06.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:06.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:06.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:06.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:06.973+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:07.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:06.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:07.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:06.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:07.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:06.838+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:07.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:06.839+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:07.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:07.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:07.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:07.221+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:07.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:07.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:07.976+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:07.977+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:07.847+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:07.847+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:07.847+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:07.847+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:08.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:08.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:08.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:08.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:09.188 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:08.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:09.188 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:08.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:09.188 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:08.967+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:09.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:08.968+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:09.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:08.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:09.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:08.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:09.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:08.871+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:09.189 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:08.872+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:09.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:09.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:09.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:09.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:09.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:09.187+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:09.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:09.188+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:10.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:09.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:10.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:09.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:10.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:09.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:10.179 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:09.975+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:10.179 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:09.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:10.179 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:09.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:10.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:09.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:10.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:09.862+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:10.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:10.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:10.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:10.190+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:10.841 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:10.546+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21907 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:11.154 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:11.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:11.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:11.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:11.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:11.022+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:11.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:11.022+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:11.155 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:10.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:11.156 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:10.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:11.156 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:10.841+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:11.156 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:10.842+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:11.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:11.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:11.154+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:11.155+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:12.136 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:12.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:12.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:12.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:12.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:11.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:11.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:12.137 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:11.822+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:12.138 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:11.822+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:12.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:12.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:12.136+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:12.136+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:13.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:12.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:13.107 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:12.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:13.107 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:12.809+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:13.107 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:12.809+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:13.107 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:13.107 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:13.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:13.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.012+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:13.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:13.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:13.106+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:13.107+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:14.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:14.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:14.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:14.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:13.987+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:14.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:13.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:14.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:13.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:14.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:13.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:14.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:13.854+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:14.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:14.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:14.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:14.131+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:15.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:14.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:15.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:14.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:15.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:14.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:15.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:14.967+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:15.095 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:14.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:15.095 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:14.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:15.095 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:14.843+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:15.095 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:14.843+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:15.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:15.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:15.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:15.094+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:15.811 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:15.549+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 21995 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:16.068 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:15.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:16.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:15.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:16.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:15.969+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:16.069 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:15.969+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:16.069 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:15.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:16.069 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:15.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:16.070 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:15.810+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:16.070 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:15.811+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:16.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:16.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:16.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:16.068+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:16.069+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:17.052 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:16.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:17.052 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:16.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:17.052 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:16.798+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:17.053 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:16.798+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:17.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:16.930+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:17.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:16.930+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:17.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:16.930+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:17.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:16.931+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:17.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:17.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:17.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:17.053+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:17.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:17.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:17.835+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:18.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:17.836+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:17.978+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:17.978+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:17.978+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:17.979+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:18.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:18.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:18.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:18.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:18.032+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:18.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:18.033+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:18.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:18.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:18.988+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:18.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:18.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:18.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:18.863+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:18.864+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:19.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:19.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:19.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:19.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:19.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:19.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:19.012+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:20.179 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:20.180 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:20.180 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:20.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:20.180 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:20.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:20.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:19.850+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:20.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:19.850+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:20.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:19.850+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:20.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:19.851+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:20.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:19.947+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:20.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:19.947+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:20.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:19.947+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:20.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:19.948+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:20.840 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:20.551+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22086 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:21.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:21.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:21.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:21.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:21.088+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:20.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:20.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:20.944+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:20.945+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:20.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:21.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:20.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:21.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:20.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:21.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:20.840+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:22.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:21.917+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:22.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:21.917+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:22.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:21.917+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:22.108 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:21.918+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:22.108 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:21.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:22.109 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:21.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:22.109 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:21.840+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:22.109 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:21.840+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:22.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:22.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:22.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:22.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:22.121+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:23.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:23.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:23.102+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:23.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:23.102+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:22.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:22.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:22.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:22.920+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:22.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:23.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:22.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:23.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:22.856+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:23.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:22.856+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:24.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:24.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:24.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:24.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:24.054+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:23.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:23.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:23.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:23.907+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:23.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:24.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:23.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:24.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:23.877+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:24.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:23.878+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:25.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:25.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:25.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:25.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:25.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:24.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:24.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:24.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:24.864+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:24.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:25.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:24.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:25.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:24.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:25.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:24.889+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:25.815 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:25.554+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22174 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:26.079 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:25.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:26.079 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:25.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:26.079 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:25.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:26.079 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:25.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:26.079 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:25.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:26.080 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:25.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:26.080 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:25.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:26.080 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:25.929+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:26.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:26.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:26.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:26.079+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:27.035 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:26.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:27.035 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:26.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:26.963+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:26.964+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:26.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:26.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:26.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:27.036 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:26.779+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:27.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:27.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:27.035+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:27.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:27.035+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:27.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:27.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:27.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:27.804+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.044+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.045+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:27.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:27.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:28.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:27.994+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:28.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:27.994+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:28.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:29.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:28.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:29.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:28.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:29.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:28.762+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:29.228 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:29.228 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:29.228 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:29.229 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:28.999+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:29.229 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:28.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:29.229 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:28.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:29.229 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:28.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:29.229 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:28.950+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:30.180 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:29.953+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:29.953+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:29.953+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:29.953+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:29.969+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:29.969+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:30.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:29.969+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:30.182 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:29.970+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:30.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:29.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:29.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:29.769+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:29.770+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:30.930 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:30.556+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22264 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:30.931 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:30.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.931 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:30.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.931 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:30.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:30.931 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:30.757+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:30.930+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:30.930+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:30.930+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:31.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:30.931+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:31.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:30.931+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:30.931+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:30.931+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:30.932+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:32.141 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:31.889+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:32.142 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:31.889+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:32.142 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:31.889+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:32.142 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:31.890+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:32.142 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:31.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:31.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:31.921+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:31.922+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:31.783+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:31.783+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:31.783+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:32.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:31.783+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:32.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:32.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:32.734+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:32.735+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:32.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:32.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:32.936+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:32.937+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:32.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:32.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:32.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:32.904+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:33.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:33.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:33.692+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:33.693+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:33.910+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:33.910+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:33.910+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:33.911+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:33.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:33.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:33.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:33.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:33.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:33.903+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:35.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:34.960+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:35.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:34.960+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:35.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:34.960+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:35.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:34.961+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:34.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:34.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:34.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:34.738+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:35.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:34.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:35.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:34.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:35.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:34.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:35.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:34.891+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:35.919 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:35.558+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22352 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:35.919 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:35.721+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.919 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:35.721+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:35.721+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:35.920 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:35.722+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:35.993+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:35.993+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:35.993+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:35.993+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:35.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:35.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:35.917+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:35.918+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:37.105 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:36.958+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:37.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:36.958+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:37.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:36.958+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:37.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:36.958+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:37.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:36.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:36.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:36.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:36.756+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:36.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:36.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:36.947+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:37.107 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:36.948+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:37.988 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:37.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.988 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:37.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.989 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:37.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:37.989 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:37.727+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:37.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:37.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:37.987+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:37.987+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:38.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:38.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:38.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:38.008+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:39.013 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:38.747+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:39.014 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:38.747+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:39.014 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:38.747+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:39.014 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:38.748+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:39.014 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:38.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:39.014 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:38.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:39.015 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:38.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:39.015 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:38.985+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:39.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.012+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.012+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:40.180 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:39.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:40.180 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:39.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:39.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:39.945+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.992+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:40.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:39.992+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:40.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:39.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:39.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:39.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:39.751+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:40.561+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22443 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:40.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:40.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:40.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:40.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:40.791+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:41.357 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:41.358 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:41.358 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:41.358 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.010+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:41.358 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:40.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:41.358 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:40.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:41.359 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:40.961+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:41.359 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:40.961+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:41.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:41.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:41.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:41.776+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:42.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:42.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:42.007+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:42.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:42.008+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:42.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:42.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:42.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.985+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:42.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:41.985+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:43.038 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:42.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:43.038 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:42.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:43.038 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:42.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:43.038 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:42.760+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:43.039 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:42.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:43.039 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:42.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:43.039 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:42.964+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:43.039 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:42.964+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:43.450 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:43.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:43.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:43.036+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:43.037+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.016+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:44.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.017+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:43.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:43.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:43.978+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:44.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:43.978+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:43.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:44.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:43.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:44.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:43.807+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:44.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:43.808+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:44.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:44.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:44.776+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:44.777+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:45.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.974+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:45.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.974+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:45.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.974+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:45.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:44.975+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:45.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:44.954+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:45.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:44.954+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:45.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:44.954+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:45.205 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:44.954+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:45.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:45.564+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22531 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:45.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:45.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:45.799+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:45.799+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:45.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:45.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:45.984+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:46.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:45.985+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:45.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:45.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:45.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:45.996+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:46.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:46.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:46.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:46.959+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:46.972+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:46.972+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:46.972+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:46.972+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:46.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:46.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:46.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:46.817+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:47.931+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:47.931+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:47.931+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:48.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:47.932+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:47.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:47.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:47.781+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:47.782+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:47.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:48.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:47.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:48.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:47.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:48.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:47.934+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:48.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:48.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:49.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:48.886+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:49.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:48.887+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:49.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:48.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:49.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:48.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:49.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:48.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:49.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:48.790+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:49.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:48.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:49.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:48.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:49.204 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:48.893+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:49.205 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:48.894+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:50.181 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:49.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:49.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:49.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:49.938+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:49.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:49.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:50.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:49.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:50.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:49.833+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:50.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:49.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:50.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:49.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:50.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:49.883+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:50.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:49.883+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:50.856 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:50.566+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22621 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:51.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:50.943+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:51.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:50.943+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:51.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:50.943+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:51.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:50.943+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:51.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:50.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:51.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:50.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:51.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:50.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:51.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:50.908+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:50.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:50.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:50.855+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:50.856+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:51.880+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:51.880+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:51.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:51.881+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:52.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:51.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:52.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:51.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:52.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:51.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:52.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:51.890+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:52.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:51.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:52.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:51.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:52.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:51.971+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:52.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:51.972+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:52.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:52.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:52.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:52.888+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:52.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:52.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:52.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:52.904+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:52.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:53.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:52.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:53.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:52.937+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:53.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:52.937+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:53.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:53.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:53.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:53.882+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:53.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:53.912+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:53.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:53.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:54.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:53.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:54.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:53.981+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:54.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:54.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:54.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:54.876+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:54.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:54.926+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:54.926+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:54.927+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:55.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:55.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:55.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:55.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:55.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:55.017+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:55.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:55.017+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:55.888 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:07:55.569+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22709 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:07:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:56.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:56.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:56.010+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:56.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:56.010+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:55.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:55.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:55.888+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:56.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:55.889+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:56.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:55.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:56.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:55.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:56.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:55.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:56.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:55.895+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:56.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:56.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:56.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:56.911+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:56.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:56.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:56.846+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:56.846+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:57.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:57.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:57.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:57.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:57.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:57.052+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:57.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:57.053+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:57.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:57.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:57.894+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:57.895+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:58.061+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:58.061+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:58.061+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:58.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:58.061+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:57.929+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:58.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:57.929+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:58.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:57.929+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:58.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:57.931+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:58.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:58.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:58.965+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:07:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:58.966+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:59.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:59.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:59.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:59.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:59.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:59.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:07:59.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:07:59.018+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:07:59.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:58.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:59.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:58.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:59.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:58.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:07:59.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:58.929+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:00.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:59.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:00.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:59.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:00.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:59.969+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:00.183 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:07:59.970+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:00.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:59.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:00.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:59.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:00.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:59.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:00.184 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:07:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:07:59.928+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:00.184 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:00.037+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:00.184 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:00.037+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:00.184 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:00.037+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:00.184 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:00.038+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:00.899 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:00.571+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22800 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:01.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:01.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:01.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:01.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:01.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:01.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:01.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:01.074+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:01.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:00.989+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:01.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:00.989+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:01.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:00.989+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:01.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:00.989+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:01.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:00.899+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:01.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:00.899+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:01.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:00.899+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:01.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:00.900+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:01.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:01.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:01.991+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:02.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:01.991+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:01.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:01.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:01.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:01.919+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:02.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:02.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:02.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:02.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:02.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:02.067+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:02.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:02.068+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:03.103+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:03.103+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:03.103+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:03.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:03.103+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:02.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:02.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:02.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:02.974+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:03.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:02.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:03.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:02.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:03.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:02.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:03.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:02.952+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:04.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:03.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:04.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:03.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:04.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:03.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:04.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:03.990+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:04.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:03.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:04.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:03.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:04.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:03.981+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:04.106 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:03.981+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:04.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:04.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:04.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:04.105+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:05.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:05.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:05.111+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:05.112+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:05.000+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:05.000+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:05.000+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:05.000+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:04.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:04.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:04.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:05.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:04.955+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:05.711 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:05.574+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22888 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:05.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:05.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:05.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:05.965+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:06.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:06.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:06.013+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:06.013+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:06.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:06.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:06.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:06.152+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:07.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:06.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:07.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:06.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:07.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:06.937+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:07.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:06.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:07.194 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:07.023+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:07.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:07.023+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:07.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:07.023+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:07.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:07.023+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:07.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:07.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:07.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:07.194+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:08.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:08.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:08.066+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:08.066+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:07.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:07.986+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:07.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:08.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:07.987+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:08.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:08.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:08.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:08.213+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:09.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:09.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:09.092+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:09.093+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:08.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:08.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:08.939+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:08.939+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:09.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:09.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:09.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:09.258+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:10.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:09.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:10.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:09.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:10.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:09.894+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:10.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:09.895+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:10.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:10.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:10.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:10.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:10.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:10.074+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:10.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:10.074+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:10.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:10.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:10.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:10.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:10.916 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:10.577+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 22978 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:10.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:10.916+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:11.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:11.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:11.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:11.078+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:11.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:11.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:11.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:11.900+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:11.900+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:12.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:12.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:12.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:12.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:12.037+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:12.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:12.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:12.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:12.331+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:13.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:13.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:13.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:13.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:13.079+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:12.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:12.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:12.948+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:13.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:12.949+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:13.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:13.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:13.350+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:13.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:14.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:14.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:14.050+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:14.050+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:13.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:13.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:13.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:13.999+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:14.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:14.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:14.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:14.397+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:15.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:15.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:15.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:15.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:15.352 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:15.059+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:15.353 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:14.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:15.353 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:14.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:15.353 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:14.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:15.353 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:14.975+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:15.602 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:15.579+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23066 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:15.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:15.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:15.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:15.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:15.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:15.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:15.602 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:15.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:16.358 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:15.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:15.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:15.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:15.968+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:16.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:16.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:16.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:16.359 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:16.100+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:16.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:16.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:16.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:16.359+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:16.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:16.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:16.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:16.921+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:17.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:17.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:17.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:17.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:17.145+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:17.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:17.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:17.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:17.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:17.946+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:17.946+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:18.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:18.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:18.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:18.194+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:18.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:18.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:18.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:18.389+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:18.901+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:18.901+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:18.901+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:18.902+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:19.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:19.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:19.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:19.146+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:19.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:19.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:19.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:19.439+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:20.183 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:19.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:19.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:19.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:19.903+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:20.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:20.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:20.184 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:20.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:20.185 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:20.133+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:20.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:20.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:20.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:20.410+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:20.943 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:20.582+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23157 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:20.942+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:20.942+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:20.942+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:20.943+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:21.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:21.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:21.101+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:21.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:21.102+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:21.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:21.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:21.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:21.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:21.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:21.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:21.910+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:21.911+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:22.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:22.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:22.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:22.104+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:22.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:22.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:22.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:22.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:23.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:23.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:23.111+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:23.112+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:22.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:22.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:22.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:22.922+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:23.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:23.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:23.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:23.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:24.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:24.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:24.065+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:24.066+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:23.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:23.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:23.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:23.876+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:24.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:24.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:24.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:24.274+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:25.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:25.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:25.077+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:25.077+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:24.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:24.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:24.831+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:24.832+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:25.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:25.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:25.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:25.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:25.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:25.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:25.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:25.301+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:25.585+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23245 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:25.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:25.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:25.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:25.821+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:26.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:26.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:26.127+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:26.127+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:26.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:26.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:26.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:26.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:27.138 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:26.778+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:26.778+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:26.778+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:26.779+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:27.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:27.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:27.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:27.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:27.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:27.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:27.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:27.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:27.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:27.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:27.793+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:27.794+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:28.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:28.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:28.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:28.164+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:28.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:28.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:28.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:28.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:29.145 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:28.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:29.145 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:28.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:29.146 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:28.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:29.146 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:28.800+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:29.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:29.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:29.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:29.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:29.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:29.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:29.145+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:29.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:29.145+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:30.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:29.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:30.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:29.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:30.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:29.816+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:30.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:29.817+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:30.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:30.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:30.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:30.240+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:30.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:30.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:30.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:30.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:30.118+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:30.855 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:30.587+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23335 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:30.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:30.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:30.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:30.855+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:31.094+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:31.094+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:31.094+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:31.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:31.095+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:31.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:31.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:31.232+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:31.233+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:32.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:31.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:31.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:31.872+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.139 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:31.873+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:32.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:32.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:32.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:32.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:32.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:32.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:32.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:32.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:32.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:32.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:32.862+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:32.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:32.863+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:33.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:33.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:33.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:33.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:33.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:33.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:33.091+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:34.098 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:33.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:34.098 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:33.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:34.098 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:33.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:34.099 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:33.835+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:34.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:34.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:34.097+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:34.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:34.098+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:34.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:34.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:34.259+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:34.260+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:35.117 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:34.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.117 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:34.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.117 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:34.847+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.118 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:34.847+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:35.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:35.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:35.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:35.224+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:35.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:35.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:35.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:35.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:35.117+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:35.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:35.590+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23423 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:35.819+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:35.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:35.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:36.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:36.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:36.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:36.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:36.140+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:36.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:36.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:36.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:36.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:37.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:36.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:36.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:36.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:36.822+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:37.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:37.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:37.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:37.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:37.160+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:37.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:37.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:37.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:37.211+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:37.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:37.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:37.773+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:37.774+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:38.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:38.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:38.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:38.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:38.206+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:38.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:38.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:38.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:38.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:38.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:38.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:38.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:38.197+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:39.170 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:38.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:39.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:38.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:39.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:38.785+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:39.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:38.786+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:39.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:39.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:39.193+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:39.193+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:39.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:39.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:39.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:39.170+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:40.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:39.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:40.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:39.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:40.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:39.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:40.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:39.816+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:40.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:40.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:40.235+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:40.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:40.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:40.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:40.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:40.156+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:40.157+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:40.849 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:40.592+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23514 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:41.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:40.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:41.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:40.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:41.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:40.849+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:41.182 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:40.850+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:41.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:41.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:41.189+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:41.190+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:41.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:41.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:41.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:41.182+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:42.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:41.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:42.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:41.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:42.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:41.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:42.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:41.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:42.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:42.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:42.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:42.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:42.166+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:42.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:42.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:42.142+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:42.143+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:43.152 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:43.153 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:43.153 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:42.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:43.153 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:42.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:43.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:43.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:43.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:43.153+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:43.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:43.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:43.213+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:43.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:43.213+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:44.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:43.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:44.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:43.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:44.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:43.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:44.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:43.805+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:44.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:44.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:44.184+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:44.185+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:44.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:44.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:44.131+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:44.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:44.132+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:45.088 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:44.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:45.088 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:44.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:45.088 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:44.804+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:45.088 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:44.805+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:45.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:45.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:45.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:45.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:45.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:45.152+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:45.152+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:45.853 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:45.594+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23602 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:46.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:46.070+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:46.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:46.070+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:46.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:46.070+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:46.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:46.071+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:46.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:45.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:46.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:45.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:46.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:45.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:46.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:45.853+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:46.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:46.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:46.146+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:47.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:47.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:47.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:47.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:47.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:47.024+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:46.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:46.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:46.902+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:46.903+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:47.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:47.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:47.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:47.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:47.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:47.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:47.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:47.132+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:48.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:47.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:48.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:47.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:48.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:47.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:48.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:47.914+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:48.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:48.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:48.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:48.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:48.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:48.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:48.173 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:48.004+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:48.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:48.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:48.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:48.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:48.171+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:49.218 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:49.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:49.218 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:49.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:49.218 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:49.002+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:49.218 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:49.002+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:49.219 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:48.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:49.219 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:48.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:49.219 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:48.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:49.219 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:48.955+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:49.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:49.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:49.216+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:49.217+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:49.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:49.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:50.186 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:49.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:50.187 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:49.926+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:50.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:50.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:50.019+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:50.187 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:50.019+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:50.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:50.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:50.200+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:50.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:50.597+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23692 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:50.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:51.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:50.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:51.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:50.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:51.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:50.957+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:51.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:51.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:51.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:51.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:51.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:51.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:51.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:51.193+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:51.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:51.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:51.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:51.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:51.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:51.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:51.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:51.052+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:52.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:52.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:52.078+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:52.078+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:51.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:51.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:51.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:51.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:52.207+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:52.207+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:53.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:53.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:53.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:53.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:53.200 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:53.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:53.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:53.016+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:53.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:53.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:53.060+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:53.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:53.061+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:53.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:53.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:53.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:53.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:54.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:54.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:54.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:54.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:54.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:54.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:54.106 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:54.058+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:54.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:54.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:54.182+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:54.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:54.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:54.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:54.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:54.104+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:54.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:54.105+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:55.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:55.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:55.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:55.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:55.187+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:55.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:55.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:55.064+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:55.065+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:55.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:55.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:55.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:55.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:55.047+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:55.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:08:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:08:55.600+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23797 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:08:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:56.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:56.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:56.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:56.016+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:56.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:56.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:56.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:56.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:56.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.007+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:56.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.008+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:56.970+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.013+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:57.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:57.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:57.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:57.126+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:58.139 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:58.139 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:58.139 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.985+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:58.139 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:57.986+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:58.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:57.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:58.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:57.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:58.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:57.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:58.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:57.962+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:58.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:58.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:58.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:58.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:58.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:58.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:58.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:08:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:08:58.984+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:58.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:58.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:58.990+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:08:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:08:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:08:58.991+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:08:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:59.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:59.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:59.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:59.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:59.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:08:59.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:08:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:08:59.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:00.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:00.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:00.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:00.006+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:00.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:00.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:00.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:00.040+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:00.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:00.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:00.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:00.175+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:00.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:00.175+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:00.603+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23887 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:01.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:01.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:01.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:01.062+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:01.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:01.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:01.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:01.186+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:01.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:01.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:01.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:01.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:01.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:01.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:01.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:01.032+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:02.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:02.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:02.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:02.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:02.074+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:02.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:02.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:02.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:02.040+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:02.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:02.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:02.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:02.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:02.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:02.164+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:03.163 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:03.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:03.163 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:03.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:03.163 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:03.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:03.163 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:03.041+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:03.164 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:03.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:03.164 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:03.056+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:03.164 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:03.057+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:03.164 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:03.058+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:03.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:03.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:03.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:03.163+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:04.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:04.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:04.148 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:04.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:04.016+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:04.017+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:04.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:04.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:04.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:04.149 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:04.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:04.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:04.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:04.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:04.148+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:05.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:05.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:05.026+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:05.028+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:05.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:05.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:05.148+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:05.148+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:05.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:05.042+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:05.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:05.042+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:05.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:05.042+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:05.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:05.043+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:05.606+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 23976 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:06.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:06.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:06.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:06.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:06.060+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:06.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:06.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:06.059+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:06.060+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:06.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:06.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:06.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:06.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:06.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:06.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:06.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:06.131+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:07.353 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:07.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:07.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:07.033+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:07.034+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:07.354 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:07.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:07.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:07.062+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:07.063+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:07.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:07.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:07.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:07.355 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:07.182+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:08.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:08.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:08.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:08.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:08.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:08.108+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:08.200 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:08.109+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:08.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:08.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:08.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:08.082+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:08.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:08.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:08.199+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:08.200+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:09.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:09.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:09.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:09.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:09.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:09.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:09.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:09.063+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:09.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:09.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:09.235+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:09.235+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:09.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:09.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:09.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:09.456 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:09.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:09.456 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:09.154+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:10.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:10.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:10.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:10.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:10.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:10.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:10.447 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:10.110+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:10.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:10.275+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:10.275+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:10.275+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:10.275+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:10.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:10.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:10.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:10.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:10.100+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:10.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:10.608+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24065 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:11.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:11.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:11.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:11.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:11.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:11.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:11.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:11.272+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:11.273+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:11.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:11.099+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:11.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:11.099+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:11.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:11.099+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:11.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:11.100+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:12.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:12.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:12.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:12.069+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:12.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:12.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:12.093+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:12.093+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:12.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:12.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:12.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:12.247+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:12.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:12.248+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:13.293 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:13.061+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:13.293 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:13.061+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:13.061+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:13.062+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:13.101+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:13.101+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:13.101+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:13.294 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:13.101+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:13.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:13.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:13.292+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:13.293+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:14.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:14.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:14.066+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:14.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:14.066+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:14.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:14.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:14.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:14.090+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:14.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:14.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:14.246+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:14.246+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:15.050+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:15.050+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:15.050+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:15.051+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:15.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:15.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:15.228+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:15.228+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:15.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:15.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:15.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:15.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:15.062+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:15.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:15.063+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:15.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:15.611+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24154 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:16.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:16.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:16.027+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:16.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:16.027+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:16.081+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:16.081+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:16.081+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:16.082+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:16.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:16.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:16.260+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:16.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:16.260+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.008+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:17.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.009+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:17.033+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:17.033+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:17.033+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:17.034+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:17.229+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:17.229+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:17.229+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:17.229+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:18.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:18.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:18.053+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:18.054+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:18.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:18.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:18.181+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:18.182+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:18.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:18.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:18.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.987+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:18.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:17.989+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:19.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:19.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:19.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:19.018+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:19.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:19.019+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:19.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:19.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:19.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.018+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:19.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.019+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:19.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:19.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:19.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:19.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:19.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:19.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.015+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.972+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:19.973+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:20.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:20.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:20.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:20.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:20.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:20.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:20.213+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:20.614+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24244 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:21.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:20.971+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:21.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:21.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:21.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:21.205+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:20.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:20.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:20.965+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:21.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:20.966+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:21.933+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:21.933+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:21.933+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:21.934+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:22.179+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:22.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:22.180+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:22.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:22.180+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:22.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:21.999+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:22.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:21.999+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:22.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:21.999+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:22.203 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:21.999+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:22.990 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:22.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.990 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:22.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.990 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:22.896+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:22.990 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:22.896+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:22.988+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:22.988+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:22.988+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:23.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:22.989+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:23.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:23.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:23.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:23.184+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:24.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:23.935+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:24.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:23.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:24.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:23.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:24.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:23.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:24.142 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:24.143 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:24.143 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.015+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:24.143 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.016+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:24.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:24.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:24.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:24.141+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:25.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:25.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:25.163+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:25.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:25.163+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:24.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:24.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:24.974+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:24.975+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:25.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.978+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:25.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.978+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:25.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.978+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:25.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:24.979+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:25.617+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24333 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:26.210 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:25.960+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:26.210 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:25.960+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:26.210 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:25.960+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:26.210 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:25.961+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:26.211 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:25.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:26.211 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:25.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:26.211 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:25.959+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:26.211 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:25.959+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:26.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:26.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:26.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:26.210+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:26.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:26.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:26.945+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:27.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:26.946+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:27.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:26.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:27.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:26.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:27.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:26.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:27.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:26.959+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:27.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:27.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:27.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:27.220+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:27.973+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:27.973+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:27.973+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:27.974+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:27.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:27.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:27.914+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:28.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:27.916+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:28.268+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:28.268+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:28.268+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:28.269+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:28.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:28.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:28.900+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:29.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:28.900+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:28.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:28.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:28.961+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:29.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:28.962+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:29.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:29.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:29.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:29.249+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:30.188 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:29.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:29.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:29.951+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:29.953+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:29.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:29.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:30.189 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:29.878+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:30.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:29.878+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:30.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:30.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:30.204+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:30.204+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:30.894 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:30.619+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24421 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:31.174 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:30.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:31.177 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:30.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:30.893+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:30.894+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:30.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:30.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:30.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:31.178 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:30.964+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:31.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:31.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:31.173+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:31.174+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:31.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:31.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:31.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:32.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:31.980+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:32.205 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:31.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:32.206 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:31.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:32.206 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:31.872+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:32.206 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:31.873+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:32.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:32.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:32.203+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:32.203+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:33.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:33.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:33.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.029+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:33.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.031+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:33.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:32.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:33.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:32.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:33.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:32.890+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:33.195 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:32.890+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:33.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:33.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:33.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:33.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:33.194+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.997+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:33.998+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:33.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:33.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:33.849+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:34.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:33.849+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:34.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:34.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:34.223+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:34.224+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:34.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:34.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:34.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:34.999+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:35.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:34.810+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:34.810+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:34.810+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:34.811+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:35.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:35.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:35.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:35.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:35.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:35.209+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:35.623 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:35.209+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:35.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:35.622+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24510 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:35.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:35.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:35.800+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:35.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:35.801+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:36.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.006+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:36.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:36.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:36.222+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:36.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:36.222+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:36.757+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:37.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:36.757+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:37.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:36.757+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:37.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:36.758+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:37.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:37.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:37.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:37.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:36.999+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:37.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:37.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:38.041 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:37.734+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:38.041 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:37.734+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:38.042 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:37.734+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:38.042 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:37.735+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:38.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:38.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:38.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:38.041+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:38.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:38.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:38.245+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:38.245+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:39.088 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:39.088 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:39.088 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:38.756+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:39.088 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:38.756+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:39.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:39.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:39.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:39.088+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:39.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:39.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:39.212+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:39.212+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:40.057 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:39.795+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:39.795+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:39.795+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.058 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:39.796+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:40.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:40.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:40.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:40.057+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:40.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:40.186+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:40.625+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24599 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:40.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:40.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:40.842+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:40.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:40.843+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:41.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:41.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:41.104+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:41.104+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:41.151+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:41.151+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:41.151+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:41.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:41.151+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:42.096 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:41.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:42.097 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:41.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:42.097 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:41.831+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:42.097 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:41.831+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:42.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:42.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:42.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:42.097+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:42.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:42.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:42.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:42.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:42.159+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:43.126 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:43.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:43.126 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:43.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:43.126 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:43.072+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:43.126 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:43.073+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:43.127 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:42.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:43.127 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:42.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:43.127 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:42.873+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:43.127 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:42.874+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:43.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:43.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:43.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:43.126+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:44.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:44.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:44.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:44.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:44.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:44.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:44.161 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:44.078+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:44.161 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:43.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:44.161 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:43.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:44.161 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:43.857+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:44.162 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:43.858+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:44.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:44.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:44.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:44.159+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:44.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:44.160+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:45.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:45.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:45.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:45.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:45.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:45.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:45.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:45.047+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:45.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:44.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:45.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:44.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:45.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:44.862+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:45.170 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:44.862+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:45.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:45.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:45.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:45.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:45.905 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:45.627+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24688 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:46.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:46.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:46.007+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:45.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:45.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:45.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:46.194 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:45.906+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:46.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:46.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:46.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:46.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:46.193+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:46.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:46.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:46.905+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:47.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:46.906+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:47.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:47.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:47.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:47.147+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:47.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:47.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:47.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:47.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:47.024+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:48.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:48.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:48.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:48.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:48.128+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:47.939+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:47.939+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:47.939+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:48.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:47.939+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:48.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:48.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:48.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:48.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:48.032+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:48.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:48.033+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:48.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:48.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:48.980+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:48.981+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:49.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:49.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:49.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:49.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:49.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:49.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:49.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:49.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:49.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:49.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:49.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:49.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:49.069+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:49.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:49.070+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:50.381 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:50.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:50.382 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:50.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:50.382 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:50.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:50.382 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:50.087+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:50.382 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:49.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:50.382 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:49.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:49.998+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:49.999+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:50.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:50.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:50.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:50.383 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:50.164+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:50.641 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:50.630+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24777 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:51.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:51.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:51.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:51.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:51.110+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:51.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:51.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:51.014+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:51.015+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:51.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:51.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:51.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:51.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:52.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:52.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:52.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:52.106+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:52.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:52.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:52.012+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:52.013+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:52.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:52.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:52.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:52.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:52.081+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:53.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:53.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:53.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:53.140+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:53.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:53.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:53.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:53.079+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:53.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:53.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:53.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:53.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:53.047+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:53.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:53.048+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:54.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:54.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:54.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:54.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:54.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:54.080+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:54.105 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:54.080+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:54.119+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:54.119+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:54.119+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:54.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:54.120+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:54.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:54.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:54.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:54.104+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:55.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:55.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:55.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:55.134+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:55.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:55.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:55.126+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:55.127+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:55.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:55.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:55.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:55.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:55.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:55.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:55.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:09:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:09:55.632+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24866 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:09:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:56.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:56.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:56.106+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:56.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:56.107+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:56.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:56.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:56.164+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:56.164+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:56.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:56.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:56.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:56.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:56.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:56.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:56.089+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:57.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:57.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:57.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:57.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:57.093+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:57.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:57.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:57.209+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:57.210+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:57.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:57.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:57.055+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:57.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:57.056+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:58.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:58.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:58.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:58.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:58.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:58.049+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:58.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:58.049+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:58.331 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:58.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:58.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:58.227+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:58.228+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:58.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:58.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:58.332 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:58.080+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:58.333 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:58.080+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.004+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:09:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.005+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:59.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:59.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:59.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:09:59.036+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:59.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:59.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:59.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:59.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:09:59.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:09:59.238+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:09:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:00.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:09:59.998+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:00.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:00.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:00.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:00.059+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:00.206+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:00.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:00.206+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:00.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:00.206+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:00.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:00.207+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:00.636+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 24955 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:01.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:01.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:01.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:01.250+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:01.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:01.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:01.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:01.108+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:01.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:01.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:01.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:01.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:01.009+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:01.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:01.010+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:02.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:02.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:02.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:02.007+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:02.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:02.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:02.149+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:02.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:02.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:02.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:02.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:02.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:03.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:03.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:03.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:03.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:03.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:03.185+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:03.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:03.185+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:03.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:03.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:03.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:03.048+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:03.049+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:03.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:03.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:03.219+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:03.371 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:03.220+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:04.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:04.085+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:04.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:04.085+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:04.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:04.085+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:04.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:04.086+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:04.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:04.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:04.193+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:04.194+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:04.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:04.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:04.243+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:04.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:04.244+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:05.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:05.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:05.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:05.115+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:05.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:05.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:05.211+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:05.212+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:05.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:05.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:05.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:05.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:05.165+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:05.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:05.165+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:05.639+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25044 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:06.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:06.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:06.114+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:06.116+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:06.194+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:06.194+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:06.194+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:06.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:06.195+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:06.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:06.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:06.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:06.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:06.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:07.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:07.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:07.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:07.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:07.353 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:07.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:07.354 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:07.183+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:07.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:07.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:07.078+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:07.354 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:07.079+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:07.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:07.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:07.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:07.355 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:07.195+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:08.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:08.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:08.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:08.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:08.408 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:08.100+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:08.409 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:08.100+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:08.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:08.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:08.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:08.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:08.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:08.141+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:08.409 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:08.142+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:08.410 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:08.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:08.410 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:08.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:08.410 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:08.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:08.410 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:08.209+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:09.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:09.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:09.186+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:09.186+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:09.071+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:09.071+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:09.071+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:09.071+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:09.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:09.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:09.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:09.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:09.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:09.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:09.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:09.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:10.058+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:10.058+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:10.058+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:10.059+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:10.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:10.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:10.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:10.237+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:10.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:10.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:10.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:10.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:10.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:10.208+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:10.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:10.209+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:10.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:10.642+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25133 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:11.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:11.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:11.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:11.220+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:11.221+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:11.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:11.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:11.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:11.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:11.082+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:11.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:11.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:11.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:11.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:11.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:11.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:12.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:12.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:12.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:12.089+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:12.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:12.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:12.240+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:12.241+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:12.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:12.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:12.215+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:12.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:12.216+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:13.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:13.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:13.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:13.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:13.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:13.183+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:13.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:13.184+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:13.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:13.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:13.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:13.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:13.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:13.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:13.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:13.105+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:13.448 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:13.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:13.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:13.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:13.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:13.210+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:13.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:13.211+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:14.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:14.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:14.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:14.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:14.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:14.059+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:14.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:14.059+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:14.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:14.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:14.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:14.241+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:14.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:14.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:14.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:14.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:15.044+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:15.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:15.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:15.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:15.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:15.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:15.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:15.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:15.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:15.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:15.106+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:15.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:15.106+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:15.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:15.645+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25221 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:16.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:16.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:16.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:16.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:16.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:16.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:16.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:16.085+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:16.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:16.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:16.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:16.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:16.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:16.057+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:16.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:16.058+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.014+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.015+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:17.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:17.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:17.246+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:17.246+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:17.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:17.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:17.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:17.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:17.057+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:18.278 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:18.278 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:17.998+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:18.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:18.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:18.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:18.279 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:18.042+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:18.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:18.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:18.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:18.279+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:19.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:18.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:19.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:18.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:19.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:18.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:19.238 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:18.968+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:19.238 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:19.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:19.239 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:19.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:19.239 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:19.030+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:19.239 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:19.030+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:19.237+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:19.238+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:20.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:20.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:20.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:20.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:20.011+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:20.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:19.981+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:20.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:19.981+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:20.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:19.981+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:20.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:19.982+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:20.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:20.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:20.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:20.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:20.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:20.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:20.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:20.209+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:20.648+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25310 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:21.234 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:20.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:21.234 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:20.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:21.234 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:20.982+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:21.234 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:20.983+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:21.235 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:21.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:21.235 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:21.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:21.235 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:21.009+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:21.235 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:21.009+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:21.233+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:21.233+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:21.233+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:21.234+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:22.254 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:21.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:22.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:21.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:22.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:21.999+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:22.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:22.000+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:22.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.044+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:22.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.044+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:22.256 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.044+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:22.256 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.044+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:22.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:22.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:22.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:22.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:23.294 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:23.302 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:23.302 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.995+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:23.302 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:22.995+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:23.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:23.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:23.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:23.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:23.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:23.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:23.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:23.048+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:23.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:23.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:24.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:24.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:24.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:24.025+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:24.090+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:24.090+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:24.090+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:24.091+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:24.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:24.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:24.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:24.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:25.348 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:25.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:25.349 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:25.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:25.349 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:25.051+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:25.349 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:25.051+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:25.349 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:25.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:25.349 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:25.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:25.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:25.138+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:25.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:25.138+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:25.652 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:25.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:25.652 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:25.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:25.652 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:25.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:25.652 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:25.348+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:25.651+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25399 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:26.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:26.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:26.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:26.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:26.123+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:26.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:26.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:26.068+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:26.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:26.068+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:26.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:26.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:26.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:26.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:26.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:26.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:26.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:26.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:27.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:27.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:27.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:27.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:27.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:27.042+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:27.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:27.042+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:27.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:27.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:27.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:27.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:27.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:27.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:27.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:27.142+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:27.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:27.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:27.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:27.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:28.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:28.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:28.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:28.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:28.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:28.054+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:28.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:28.054+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:28.324 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:28.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:28.324 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:28.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:28.324 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:28.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:28.324 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:28.132+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:28.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:28.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:28.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:28.323+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:29.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:29.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.007+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.008+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:29.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:29.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:29.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:29.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:29.099+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:29.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:29.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:29.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:29.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:30.317 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:30.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:30.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.990+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:30.318 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:29.991+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:30.318 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:30.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:30.318 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:30.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:30.319 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:30.143+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:30.319 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:30.144+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:30.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:30.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:30.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:30.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:30.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:30.410+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:30.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:30.411+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:30.654+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25488 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:31.447 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:31.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:31.447 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:31.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:31.447 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:31.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:31.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:31.097+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:31.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:31.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:31.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:31.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:31.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:31.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:31.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:31.024+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:31.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:31.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:31.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:31.447+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:32.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:32.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:32.059+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:32.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:32.060+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:32.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:32.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:32.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:32.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:32.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:32.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:32.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:32.117+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:32.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:32.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:32.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:32.483+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:33.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:33.063+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:33.447 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:33.063+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:33.063+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:33.064+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:33.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:33.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:33.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:33.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:33.149+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:33.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:33.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:33.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:33.447+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:34.131 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:34.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:34.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:34.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:34.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:34.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:34.132 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:34.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:34.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:34.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:34.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:34.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:34.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:34.131+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:34.418 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:34.132+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:34.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:34.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:34.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:34.418+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:35.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:35.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:35.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:35.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:35.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:35.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:35.448 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:35.169+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:35.448 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:35.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:35.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:35.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:35.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:35.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:35.449 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:35.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:35.701 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:35.657+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25577 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:35.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:35.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:35.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:35.448+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:36.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:36.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:36.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:36.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:36.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:36.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:36.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:36.118+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:36.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:36.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:36.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:36.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:36.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:36.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:36.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:36.130+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:36.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:36.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:36.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:36.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:36.415+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:37.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:37.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:37.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:37.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:37.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:37.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:37.426 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:37.096+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:37.427 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:37.427 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:37.427 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:37.427 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:37.090+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:37.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:37.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:37.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:37.426+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:38.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:38.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:38.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:38.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:38.060+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:38.060+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:38.053+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:38.053+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:38.053+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:38.404 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:38.053+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:38.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:38.777 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:38.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:38.777 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:38.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:38.777 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:38.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:39.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:39.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:39.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.010+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:39.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.010+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:39.034+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:39.034+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:39.034+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:39.035+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:39.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:39.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:39.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:39.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:40.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:40.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:40.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.024+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:40.368 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.025+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:40.368 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:40.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:40.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:40.369 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:39.971+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:40.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:40.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:40.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:40.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:40.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:40.367+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:40.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:40.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:40.929 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:40.660+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25667 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:40.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:40.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:40.929+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:40.929+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:40.995+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:41.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:41.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:41.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:41.404+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:41.913+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:41.913+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:41.913+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:42.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:41.914+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:42.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:41.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:42.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:41.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:42.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:41.989+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:42.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:41.990+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:42.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:42.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:42.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:42.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:42.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:42.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:42.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:42.882+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:43.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:43.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:43.010+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:43.011+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:43.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:43.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:43.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:43.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:43.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:43.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:43.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:43.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:43.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:43.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:43.876+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:43.877+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:44.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:44.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:44.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:44.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:44.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:44.021+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:44.403 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:44.022+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:44.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:44.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:44.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:44.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:44.849+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:44.849+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:44.849+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:44.849+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:45.051+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:45.051+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:45.051+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:45.052+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:45.672 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:45.663+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25756 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:45.672 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:45.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:45.672 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:45.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:45.672 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:45.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:45.672 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:45.415+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:45.867+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:45.867+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:45.867+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:45.867+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:46.373 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:46.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:46.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.006+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:46.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.007+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:46.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:46.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:46.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:46.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:46.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:46.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:46.855+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:47.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:46.855+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.966+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:46.966+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:47.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:47.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:47.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:47.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:47.383+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:47.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:47.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:47.957+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:47.958+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:47.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:47.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:48.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:47.890+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:48.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:47.891+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:48.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:48.353+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:48.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:48.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:48.940+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:49.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:48.941+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:48.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:48.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:48.907+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:49.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:48.908+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:49.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:49.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:49.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:49.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:49.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:50.194 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:49.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:50.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:49.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:50.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:49.954+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:50.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:49.954+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:50.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:49.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:49.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:49.868+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:49.868+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:50.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:50.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:50.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:50.389+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:50.666+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25845 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:50.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:50.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:50.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:50.876+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:51.358 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:50.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:51.358 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:50.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:51.358 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:50.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:51.358 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:50.957+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:51.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:51.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:51.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:51.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:51.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:51.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:51.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:52.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:51.859+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:52.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:51.859+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:51.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:51.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:51.955+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:51.955+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:52.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:52.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:52.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:52.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:52.349+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:53.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:52.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:52.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:52.875+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:53.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:52.875+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:53.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:52.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:53.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:52.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:53.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:52.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:53.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:52.959+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:53.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:53.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:53.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:53.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:53.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:53.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:53.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:53.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:53.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:53.917+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:53.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:53.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:53.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:54.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:53.904+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:54.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:54.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:54.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:54.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:54.900+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:55.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:54.900+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:54.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:54.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:54.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:54.871+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:55.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:55.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:55.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:55.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:55.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:55.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:55.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:55.216+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:55.938 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:10:55.669+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 25934 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:10:55.938 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:55.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.939 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:55.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.939 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:55.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:55.939 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:55.837+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:55.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:55.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:55.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:56.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:55.938+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:56.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:56.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:56.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:56.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:56.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:56.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:56.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:56.831+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:56.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:56.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:56.912+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:57.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:56.912+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:57.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:57.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:57.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:57.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:57.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:57.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:57.885+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:58.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:57.886+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:57.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:57.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:57.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:57.872+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:58.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:58.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:58.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:58.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:58.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:58.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:58.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:58.869+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:10:59.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:58.869+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:58.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:58.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:58.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:10:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:58.835+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:10:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:59.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:59.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:59.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:59.241+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:10:59.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:10:59.242+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:59.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:59.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:59.865+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:10:59.866+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:59.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.195 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:59.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:59.811+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:10:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:10:59.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:00.255+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:00.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:00.672+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26024 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:00.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:00.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:00.861+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:00.861+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:00.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:00.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:00.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:00.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:00.802+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:01.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:01.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:01.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:01.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:01.817+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:01.817+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:01.817+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:01.818+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:02.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:01.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:02.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:01.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:02.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:01.874+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:02.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:01.875+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:02.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:02.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:02.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:02.307+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:02.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:02.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:02.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:02.853+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:02.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:02.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:02.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:03.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:02.896+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:03.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:03.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:03.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:03.292+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:03.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:03.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:03.830+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:04.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:03.830+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:03.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:03.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:03.881+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:03.881+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:04.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:04.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:04.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:04.306+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:04.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:04.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:04.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:04.855+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:04.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:04.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:04.889+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:04.889+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:05.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:05.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:05.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:05.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:05.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:05.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:05.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:05.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:05.674+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26113 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:05.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:05.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:05.863+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:05.863+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:05.842+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:05.842+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:05.842+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:05.842+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:06.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:06.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:06.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:06.353+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:06.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:06.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:06.816+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:06.816+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:06.835+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:07.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:07.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:07.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:07.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:07.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:07.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:08.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:07.774+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:08.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:07.774+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:08.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:07.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:08.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:07.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:08.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:07.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:08.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:07.848+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:08.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:08.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:08.348+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:08.349+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:08.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:08.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:08.821+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:08.821+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:08.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:08.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:08.875+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:08.875+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:09.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:09.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:09.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:09.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:10.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:09.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:09.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:09.873+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:09.873+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:10.196 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:09.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:09.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:09.853+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:09.853+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:10.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:10.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:10.417+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:10.418+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:10.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:10.678+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26202 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:10.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:10.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:10.898+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:10.899+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:10.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:10.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:10.833+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:10.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:10.834+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:11.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:11.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:11.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:11.397+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:11.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:11.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:11.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:12.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:11.932+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:12.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:11.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:12.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:11.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:12.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:11.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:12.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:11.849+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:12.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:12.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:12.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:12.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:12.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:12.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:12.903+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:13.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:12.904+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:12.876+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:12.876+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:12.876+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:13.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:12.877+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:13.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:13.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:13.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:13.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:13.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:13.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:13.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:14.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:13.919+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:13.897+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:13.897+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:13.897+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:14.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:13.897+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:14.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:14.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:14.381+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:14.381+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:14.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:14.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:14.848+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:14.849+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:14.932+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:14.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:14.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:15.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:14.933+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:15.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:15.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:15.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:15.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:15.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:15.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:15.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:15.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:15.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:15.681+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26291 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:15.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:15.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:15.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:15.847+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:16.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:15.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:16.406 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:15.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:16.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:15.971+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:16.407 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:15.971+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:16.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:16.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:16.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:16.407+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:16.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:16.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:16.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:16.807+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:16.940+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:16.940+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:16.940+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:17.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:16.940+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:17.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:17.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:17.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:17.426+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:17.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:17.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:17.802+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:18.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:17.803+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:17.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:17.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:17.904+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:18.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:17.904+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:18.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:18.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:18.426+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:18.427+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:18.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:18.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:18.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:19.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:18.800+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:18.926+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:18.926+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:18.926+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:19.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:18.926+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:19.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:19.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:19.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:19.405+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:20.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:19.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:19.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:19.837+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.197 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:19.838+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:20.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:19.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:19.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:19.936+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:19.937+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:20.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:20.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:20.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:20.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:20.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:20.684+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26381 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:20.895+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:20.895+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:20.895+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:20.895+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:20.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:20.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:20.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:20.839+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:21.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:21.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:21.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:21.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:22.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:21.902+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:22.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:21.902+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:21.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:21.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:21.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:21.855+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:22.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:22.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:22.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:22.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:22.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:22.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:22.933+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:23.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:22.933+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:22.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:22.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:22.815+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:22.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:23.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:23.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:23.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:23.346+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:23.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:23.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:23.896+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:24.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:23.897+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:23.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:23.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:23.784+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:23.784+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:24.354+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:24.943+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:24.944+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:24.812+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:24.812+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:24.812+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:24.812+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:25.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:25.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:25.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:25.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:25.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:25.687+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26470 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:25.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:25.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:25.919+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:25.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:25.919+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:25.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:25.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.953 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:25.796+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:25.953 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:25.797+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:26.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:26.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:26.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:26.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:26.319+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:26.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:26.319+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:26.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:26.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:26.829+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:27.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:26.830+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:26.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:26.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:26.937+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:27.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:26.937+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:27.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:27.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:27.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:27.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:27.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:27.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:27.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:27.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:27.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:27.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:27.973+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:28.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:27.974+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:28.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:28.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:28.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:28.330+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:29.043 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:28.766+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:29.043 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:28.766+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:29.043 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:28.766+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:29.043 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:28.766+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:29.043 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:28.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:29.044 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:28.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:29.044 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:28.975+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:29.044 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:28.976+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:29.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:29.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:30.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:30.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:30.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:30.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:30.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:30.005+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:30.197 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:30.005+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:30.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:29.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:29.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:29.768+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:29.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:30.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:30.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:30.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:30.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:30.317+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:30.690+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26559 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:30.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:30.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:30.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:30.745+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:31.355 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:31.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:31.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:31.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:31.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:31.020+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:31.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:31.020+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:31.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:31.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:31.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:31.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:32.024 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:31.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:32.025 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:31.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:32.025 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:31.791+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:32.025 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:31.792+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:32.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:32.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:32.386 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.024+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:32.387 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.024+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:32.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:32.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:32.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:32.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:32.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:32.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:32.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:32.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.982+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:33.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:32.983+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:33.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:33.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:33.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:33.365+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:34.083 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:33.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:34.083 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:33.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:34.083 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:33.764+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:34.083 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:33.765+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:34.083 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:33.974+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:34.084 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:33.974+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:34.084 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:33.974+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:34.084 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:33.974+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:34.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:34.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:34.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:34.416+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:34.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:34.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:34.948+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:35.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:34.949+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:34.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:34.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:34.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:34.725+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:35.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:35.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:35.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:35.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:35.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:35.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:35.685 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:35.371+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:35.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:35.693+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26648 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:35.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:35.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:35.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:35.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:35.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:36.370 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:35.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:36.371 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:35.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:36.371 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:35.957+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:36.371 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:35.957+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:36.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:36.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:36.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:36.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:36.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:36.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:36.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:36.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:36.680+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:36.680+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:36.680+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:36.681+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:37.366 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:36.968+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:37.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:36.968+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:37.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:36.968+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:37.367 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:36.968+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:37.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:37.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:37.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:37.367+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:37.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:37.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:38.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:37.730+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:38.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:37.731+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:38.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:37.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:38.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:37.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:38.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:37.934+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:38.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:37.935+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:38.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:38.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:38.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:38.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:39.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:38.959+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:39.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:38.959+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:39.123 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:38.959+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:39.124 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:38.959+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:39.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:38.750+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:39.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:38.750+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:39.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:38.750+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:39.124 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:38.750+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:39.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:39.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:39.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:39.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:39.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:40.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:39.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:39.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:39.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:39.710+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:40.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:39.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:40.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:39.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:40.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:39.956+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:40.199 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:39.957+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:40.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:40.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:40.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:40.337+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:40.949 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:40.696+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26738 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:40.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:40.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:40.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:40.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:40.950 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:40.668+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:40.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:40.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:41.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:40.949+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:41.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:40.949+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:41.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:41.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:41.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:41.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:41.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:41.366+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:41.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:41.367+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:41.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:41.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:41.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:41.678+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:42.335 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:41.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:42.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:41.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:42.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:41.993+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:42.336 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:41.993+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:42.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:42.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:42.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:42.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:42.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:42.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:42.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:42.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:42.674+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:43.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:42.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:43.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:42.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:43.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:42.998+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:43.323 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:42.998+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:43.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:43.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:43.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:43.323+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:43.984 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:43.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:43.984 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:43.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:43.984 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:43.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:43.985 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:43.710+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:44.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:43.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:44.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:43.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:44.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:43.983+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:44.363 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:43.984+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:44.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:44.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:44.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:44.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:44.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:44.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:44.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:44.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:44.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:44.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:44.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:44.692+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:45.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:45.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:45.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:45.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:45.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:45.008+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:45.315 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:45.009+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:45.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:45.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:45.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:45.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:45.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:45.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:45.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:45.315+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:45.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:45.699+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26827 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:45.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:45.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:45.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:45.665+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:45.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:45.666+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:46.345 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:46.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:46.346 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:46.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:46.346 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:46.006+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:46.346 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:46.007+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:46.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:46.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:46.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:46.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:46.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:47.041 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:46.711+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.041 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:46.711+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.041 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:46.711+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.042 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:46.712+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:47.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:47.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:47.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:47.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:47.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:47.040+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:47.391 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:47.041+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:47.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:47.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:47.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:47.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:47.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:47.390+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:47.692 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:47.391+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:47.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:47.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:47.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:47.692+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:48.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:48.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:48.356 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:48.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:48.357 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:48.036+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:48.357 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:48.036+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:48.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:48.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:48.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:48.650 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:48.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:48.649+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:48.649+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:48.649+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:48.650+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:49.062+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:49.062+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:49.062+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:49.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:49.062+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:49.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:49.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:49.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:49.311+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:49.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:49.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:49.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:49.659+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:50.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:50.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:50.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:50.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:50.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:50.049+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:50.309 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:50.050+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:50.617 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:50.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:50.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:50.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:50.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:50.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:50.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:50.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:50.702+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 26916 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:50.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:50.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:50.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:50.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:50.617+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:50.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:50.618+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:51.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:51.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:51.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:51.344+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:51.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:51.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:51.098+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:51.099+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:51.657+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:51.657+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:51.657+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:51.658+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:52.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:52.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:52.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:52.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:52.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:52.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:52.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:52.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:52.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:52.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:52.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:52.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:52.674+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:53.101 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:53.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:53.101 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:53.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:53.101 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:53.099+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:53.102 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:53.099+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:53.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:53.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:53.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:53.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:53.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:53.352+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:53.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:53.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:53.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:53.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:53.668+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:53.669+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:54.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:54.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:54.323+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:54.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:54.324+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:54.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:54.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:54.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:54.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:54.118+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:54.633+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:54.634+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:54.634+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:54.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:54.634+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:55.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:55.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:55.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:55.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:55.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:55.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:55.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:55.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:55.488 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:55.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:55.489 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:55.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:55.489 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:55.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:55.489 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:55.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:55.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:11:55.705+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27005 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:11:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:55.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:55.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:55.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:55.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:55.673+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:55.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:55.674+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:56.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:56.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:56.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:56.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:56.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:56.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:56.123+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:56.124+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:57.145 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:56.716+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:57.145 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:56.716+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:57.146 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:56.716+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:57.146 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:56.716+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:57.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:57.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:57.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:57.144+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:57.145+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:58.169 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:57.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:58.170 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:57.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:58.170 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:57.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:58.170 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:57.756+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:58.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:58.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:58.169+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:58.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:58.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:58.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:58.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:58.301+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:59.192 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:58.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:59.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:58.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:59.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:58.774+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:11:59.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:58.775+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:59.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:59.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:59.269+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:11:59.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:11:59.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:11:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:59.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:59.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:59.192+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:11:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:11:59.192+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:00.198 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:59.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:00.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:59.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:00.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:59.806+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:00.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:11:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:11:59.807+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:00.210+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:00.210+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:00.210+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:00.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:00.210+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:00.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:00.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:00.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:00.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:01.171 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:00.708+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27095 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:01.171 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:00.828+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:01.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:00.828+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:01.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:00.828+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:01.172 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:00.829+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:01.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:01.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:01.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:01.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:01.226+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:01.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:01.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:01.170+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:01.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:01.170+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:02.141 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:01.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:02.141 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:01.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:02.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:01.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:02.142 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:01.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:02.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:02.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:02.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:02.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:02.141+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:02.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:02.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:02.212+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:02.213+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:03.114 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:02.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:03.115 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:02.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:03.115 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:02.790+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:03.115 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:02.790+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:03.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:03.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:03.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:03.114+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:03.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:03.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:03.186+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:03.187+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:04.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:03.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:04.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:03.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:04.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:03.825+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:04.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:03.825+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:04.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:04.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:04.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:04.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:04.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:04.196+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:04.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:04.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:04.108+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:04.109+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:05.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:04.795+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:05.150 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:04.795+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:05.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:04.795+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:05.151 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:04.795+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:05.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:05.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:05.149+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:05.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:05.150+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:05.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:05.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:05.238+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:05.239+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:06.139 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:05.711+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27183 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:06.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:05.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:06.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:05.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:06.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:05.803+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:06.140 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:05.804+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:06.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:06.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:06.139+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:06.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:06.139+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:06.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:06.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:06.208+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:06.208+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:07.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:07.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:07.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:06.835+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:07.143 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:06.836+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:07.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:07.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:07.142+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:07.143+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:07.169+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:07.169+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:07.169+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:07.170+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:08.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:07.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:08.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:07.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:08.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:07.797+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:08.102 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:07.797+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:08.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:08.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:08.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:08.130+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:08.130+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:08.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:08.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:08.100+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:08.101+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:09.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:08.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:09.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:08.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:09.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:08.779+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:09.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:08.780+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:09.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:09.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:09.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:09.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:09.361 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:09.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:09.362 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:09.105+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:09.362 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:09.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:09.362 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:09.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:09.362 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:09.143+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:09.362 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:09.144+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:10.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:09.732+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:10.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:09.732+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:10.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:09.732+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:10.130 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:09.732+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:10.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:10.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:10.128+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:10.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:10.129+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:10.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:10.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:10.173+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:10.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:10.173+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:11.128 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:10.715+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27273 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:11.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:10.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:10.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:10.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.129 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:10.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:11.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:11.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:11.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:11.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:11.179+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:11.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:11.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:11.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:11.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:11.129+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:11.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:11.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:11.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:11.746+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:12.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:12.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:12.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:12.155+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:12.155+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:12.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:12.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:12.220+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:12.221+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:13.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:12.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:12.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:12.708+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.160 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:12.709+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:13.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:13.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:13.158+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:13.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:13.159+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:13.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:13.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:13.200+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:13.201+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:13.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:13.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:13.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:13.670+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:14.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:14.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:14.401 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:14.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:14.402 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:14.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:14.402 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:14.225+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:14.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:14.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:14.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:14.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:14.402 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:14.140+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:14.403 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:14.140+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:14.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:14.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:14.645+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:14.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:14.645+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:15.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:15.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:15.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:15.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:15.263+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:15.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:15.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:15.146+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:15.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:15.147+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:15.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:15.717+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27361 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:15.612+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:15.612+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:15.612+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:15.612+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:16.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:16.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:16.229+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:16.229+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:16.119+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:16.119+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:16.119+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:16.119+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:16.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:16.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:16.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:16.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:16.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:16.603+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:17.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:17.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:17.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:17.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:17.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:17.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:17.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:17.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:17.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:17.072+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:17.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:17.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:17.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:17.611+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:17.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:17.611+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:18.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:18.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:18.149+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:18.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:18.150+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:18.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:18.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:18.090+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:18.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:18.090+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:18.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:18.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:18.641+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:18.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:18.641+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:19.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:19.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:19.445 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:19.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:19.107+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:19.107+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:19.122+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:19.122+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:19.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:19.446 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:19.123+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:19.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:19.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:19.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:19.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:19.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:20.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:20.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:20.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:20.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:20.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:20.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:20.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:20.135+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:20.135+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:20.721+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27452 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:20.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:20.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:20.685+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:20.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:21.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:21.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:21.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:21.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:21.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:21.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:21.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:21.159+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:21.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:21.159+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:21.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:21.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:21.679+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:21.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:21.680+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:22.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:22.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:22.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:22.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:22.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:22.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:22.162+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:22.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:22.196+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:22.197+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:22.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:22.197+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:22.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:22.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:22.675+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:22.676+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:23.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:23.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:23.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:23.120+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:23.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:23.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:23.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:23.152+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:23.153+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:23.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:23.153+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:23.631+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:23.631+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:23.631+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:23.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:23.632+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:24.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:24.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:24.139+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:24.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:24.139+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:24.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:24.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:24.105+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:24.105+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:24.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:24.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:24.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:24.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:24.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:24.644+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:25.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:25.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:25.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:25.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:25.095+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:25.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:25.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:25.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:25.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:25.080+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:25.724+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27539 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:25.613+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:25.613+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:25.613+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:25.613+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:26.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:26.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:26.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:26.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:26.039+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:26.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:26.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:26.120+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:26.120+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:26.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:26.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:26.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:26.637+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:26.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:26.638+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:27.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:27.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:27.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:27.039+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:27.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:27.040+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:27.151+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:27.151+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:27.151+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:27.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:27.152+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:27.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:27.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:27.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:27.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:27.604+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:28.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:28.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:28.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:28.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:28.071+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:28.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:28.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:28.109+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:28.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:28.109+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:28.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:28.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:28.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:28.603+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:28.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:28.603+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:29.045+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:29.045+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:29.045+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:29.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:29.046+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:29.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:29.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:29.088+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:29.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:29.089+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:29.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:29.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:29.580+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:29.581+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:30.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:30.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:30.070+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:30.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:30.070+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:30.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:30.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:30.096+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:30.097+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:30.853 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:30.728+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27629 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:30.853 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:30.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:30.853 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:30.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:30.853 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:30.569+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:30.853 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:30.570+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:31.117 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:31.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:31.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:31.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:31.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:31.048+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:31.118 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:31.049+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:31.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:31.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:31.117+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:31.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:31.117+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:31.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:31.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:31.584+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:31.585+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:32.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:32.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:32.075+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:32.076+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:32.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:32.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:32.072+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:32.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:32.073+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:32.598+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:32.598+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:32.598+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:32.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:32.599+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:33.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:33.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:33.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:33.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:33.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:33.047+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:33.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:33.047+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:33.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:33.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:33.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:33.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:33.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:33.041+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:33.455 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:33.042+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:33.618+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:33.618+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:33.618+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:33.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:33.619+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:34.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:34.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:34.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:34.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:34.058+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:34.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:34.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:34.003+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:34.004+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:34.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:34.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:34.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:34.605+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:34.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:34.605+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:35.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:35.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:35.058+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:35.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:35.058+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.009+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.010+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:35.854 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:35.731+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27716 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:35.854 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:35.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:35.854 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:35.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:35.855 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:35.597+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:35.855 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:35.597+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.986+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.986+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.986+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:36.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:35.986+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:36.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:36.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:36.086+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:36.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:36.086+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:36.599+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:36.600+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:37.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:37.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:37.005+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:37.006+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:37.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:37.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:37.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:37.090+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:37.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:37.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:37.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:37.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:37.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:38.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:38.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:38.049+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:38.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:38.050+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:38.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:38.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:38.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:38.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:38.134+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:38.653+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:38.654+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:38.654+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:38.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:38.654+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:39.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:39.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:39.074+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:39.075+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:39.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:39.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:39.088+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:39.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:39.088+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:39.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:39.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:39.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:39.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:39.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:39.684+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:39.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:39.685+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.027+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.028+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:40.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:40.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:40.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:40.138+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:40.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:40.138+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:40.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:40.734+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27807 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:40.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:40.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:40.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:40.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:40.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:40.677+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:40.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:40.677+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.992+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:41.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.992+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:41.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.992+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:41.485 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:40.993+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:41.486 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:41.486 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:41.486 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:41.162+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:41.486 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:41.162+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:41.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:41.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:41.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:41.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:41.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:41.651+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:42.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:42.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:42.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:42.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:42.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:42.028+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:42.414 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:42.029+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:42.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:42.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:42.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:42.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:42.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:42.133+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:42.415 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:42.134+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:42.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:42.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:42.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:42.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:42.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:42.607+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:43.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:43.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:43.085+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:43.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:43.085+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:43.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:43.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:43.023+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:43.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:43.024+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:43.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:43.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:43.564+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:43.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:43.564+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:44.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:44.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:44.057+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:44.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:44.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:44.073+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:44.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:44.074+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:44.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:44.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:44.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:44.538+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:44.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:44.539+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:45.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:45.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:45.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:45.073+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:45.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:45.073+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:45.116+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:45.116+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:45.116+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:45.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:45.117+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:45.861 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:45.737+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27895 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:45.861 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:45.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:45.862 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:45.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:45.862 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:45.512+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:45.862 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:45.513+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:46.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:46.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:46.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:46.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:46.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:46.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:46.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:46.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:46.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:46.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:46.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:46.078+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:46.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:46.079+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:46.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:46.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:46.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:46.515+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:46.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:46.516+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:47.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:47.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:47.134+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:47.135+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:47.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:47.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:47.091+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:47.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:47.091+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:47.471+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:47.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:47.472+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:47.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:47.472+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:47.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:47.473+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:48.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:48.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:48.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:48.097+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:48.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:48.097+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:48.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:48.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:48.137+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:48.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:48.137+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:48.479+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:48.479+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:48.479+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:48.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:48.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:49.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:49.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:49.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:49.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:49.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:49.058+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:49.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:49.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:49.164+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:49.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:49.165+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:49.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:49.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:49.511+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:49.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:49.512+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:50.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:50.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:50.054+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:50.054+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:50.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:50.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:50.171+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:50.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:50.172+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:50.741 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:50.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:50.741 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:50.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:50.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:50.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:50.742 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:50.466+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:51.058 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:50.740+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 27985 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:51.057+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:51.057+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:51.057+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:51.058+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:51.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:51.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:51.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:51.206+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:51.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:51.206+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:51.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:51.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:51.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:51.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:51.492+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:51.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:51.493+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:52.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:52.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:52.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:52.058+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:52.059+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:52.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:52.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:52.211+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:52.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:52.211+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:52.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:52.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:52.465+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:52.466+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:53.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:53.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:53.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:53.030+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:53.031+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:53.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:53.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:53.248+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:53.249+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:53.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:53.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:53.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:53.482+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:53.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:53.482+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:54.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:54.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:54.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:54.069+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:54.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:54.070+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:54.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:54.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:54.296+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:54.297+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:54.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:54.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:54.514+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:54.514+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:55.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:55.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:55.322 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:55.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:55.323 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:55.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:55.323 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:55.067+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:55.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:55.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:55.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:55.495+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:55.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:55.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:55.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:55.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:55.322+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:55.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:55.322+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:56.095 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:12:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:12:55.744+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28073 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:12:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:56.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:56.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:56.095+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:56.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:56.096+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:56.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:56.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:56.301+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:56.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:56.301+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:56.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:56.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:56.528+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:56.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:56.529+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:57.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:57.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:57.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:57.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:57.267+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:57.267+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:57.055+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:57.055+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:57.055+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:57.055+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:57.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:57.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:57.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:57.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:57.485+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:57.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:57.486+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:58.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:58.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:58.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:58.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:58.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:58.256+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:58.076+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:58.076+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:58.076+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:58.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:58.076+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:58.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:58.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:58.498+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:58.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:58.498+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:59.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:59.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:59.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:59.226+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:12:59.227+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:59.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:59.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:59.124+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:12:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:12:59.125+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:12:59.769 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:59.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:59.769 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:59.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:59.770 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:59.503+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:12:59.770 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:12:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:12:59.504+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:00.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:00.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:00.123+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:00.123+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:00.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:00.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:00.178+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:00.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:00.179+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:00.748 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:00.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:00.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:00.455+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:00.749 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:00.456+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:01.087 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:00.747+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28164 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:01.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:01.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:01.169+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:01.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:01.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:01.086+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:01.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:01.087+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:01.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:01.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:01.476+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:01.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:01.476+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:02.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:02.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:02.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:02.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:02.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:02.063+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:02.449 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:02.064+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:02.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:02.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:02.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:02.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:02.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:02.125+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:02.450 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:02.125+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:02.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:02.448+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:02.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:02.448+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:02.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:02.448+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:02.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:02.449+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:03.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:03.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:03.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:03.116+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:03.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:03.116+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:03.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:03.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:03.082+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:03.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:03.083+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:03.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:03.850 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:03.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:03.496+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:03.497+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:04.135 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:04.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:04.136 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:04.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:04.136 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:04.071+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:04.136 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:04.072+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:04.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:04.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:04.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:04.135+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:04.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:04.135+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:04.812 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:04.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:04.813 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:04.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:04.813 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:04.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:04.813 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:04.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:05.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:05.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:05.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:05.119+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:05.120+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:05.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:05.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:05.115+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:05.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:05.115+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:05.751 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:05.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:05.752 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:05.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:05.752 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:05.499+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:05.752 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:05.500+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:06.013 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:05.751+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28252 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:06.307 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:06.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:06.307 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:06.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:06.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:06.130+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:06.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:06.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:06.091+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:06.308 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:06.092+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:06.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:06.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:06.534+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:06.535+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:07.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:07.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:07.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:07.089+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:07.089+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:07.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:07.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:07.094+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:07.095+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:07.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:07.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:07.539+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:07.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:07.539+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:08.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:08.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:08.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:08.126+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:08.127+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:08.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:08.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:08.065+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:08.066+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:08.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:08.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:08.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:08.553+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:08.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:08.554+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:09.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:09.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:09.079+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:09.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:09.079+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:09.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:09.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:09.039+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:09.040+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:09.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:09.852 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:09.852 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:09.566+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:09.852 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:09.566+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:10.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:10.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:10.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:10.043+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:10.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:10.044+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:10.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:10.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:10.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:10.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:10.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:10.076+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:10.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:10.076+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:10.878 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:10.754+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28342 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:10.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:10.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:10.878 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:10.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:10.879 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:10.607+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:10.879 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:10.608+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:11.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:11.201 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:11.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:11.087+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:11.202 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:11.088+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:11.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:11.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:11.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:11.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:11.203 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:11.058+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:11.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:11.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:11.628+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:11.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:11.629+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:12.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:12.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:12.121+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:12.122+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:12.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:12.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:12.104+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:12.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:12.104+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:12.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:12.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:12.630+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:12.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:12.631+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:13.162+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:13.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:13.162+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:13.162+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:13.163+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:13.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:13.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:13.057+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:13.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:13.058+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:13.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:13.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:13.629+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:13.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:13.630+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:14.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:14.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:14.145+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:14.146+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:14.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:14.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:14.081+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:14.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:14.081+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:14.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:14.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:14.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:14.650+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:14.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:14.651+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:15.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:15.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:15.129+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:15.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:15.129+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:15.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:15.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:15.176+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:15.176+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:15.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:15.757+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28430 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:15.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:15.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:15.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:15.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:15.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:15.694+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:16.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:16.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:16.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:16.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:16.214+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:16.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:16.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:16.121+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:16.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:16.121+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:17.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:16.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:16.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:16.727+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:16.728+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:17.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:17.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:17.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:17.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:17.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:17.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:17.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:17.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:17.425 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:17.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:17.426 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:17.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:17.426 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:17.118+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:17.426 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:17.119+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:17.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:17.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:17.720+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:17.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:17.721+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:18.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:18.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:18.114+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:18.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:18.115+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:18.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:18.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:18.254+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:18.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:18.255+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:19.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:18.718+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:18.718+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:18.718+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.155 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:18.719+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:19.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:19.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:19.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:19.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:19.274+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:19.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:19.154+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:19.154+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:19.154+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:19.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:19.155+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:19.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:19.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:19.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:19.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:19.929 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:19.679+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:20.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:20.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:20.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:20.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:20.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:20.168+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:20.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:20.168+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:20.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:20.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:20.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:20.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:20.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:20.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:20.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:20.761+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28521 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:20.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:20.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:20.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:20.643+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:20.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:20.644+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:21.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:21.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:21.256+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:21.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:21.256+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:21.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:21.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:21.131+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:21.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:21.131+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:21.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:21.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:21.691+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:21.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:21.692+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:22.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:22.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:22.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:22.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:22.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:22.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:22.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:22.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:22.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:22.453 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:22.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:22.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:22.177+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:22.454 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:22.177+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:22.654+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:22.654+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:22.654+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:22.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:22.656+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:23.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:23.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:23.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:23.333+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:23.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:23.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:23.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:23.219+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:23.661 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:23.219+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:23.658+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:23.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:23.659+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:23.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:23.659+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:24.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:24.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:24.340+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:24.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:24.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:24.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:24.256+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:24.257+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:24.968 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:24.700+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:24.969 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:24.700+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:24.969 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:24.700+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:24.969 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:24.701+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:25.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:25.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:25.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:25.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:25.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:25.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:25.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:25.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:25.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:25.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:25.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:25.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:25.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:25.261+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:25.700 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:25.261+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:25.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:25.763+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28609 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:25.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:25.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:25.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:25.697+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:25.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:25.698+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:26.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:26.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:26.344+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:26.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:26.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:26.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:26.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:26.255+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:26.255+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:26.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:26.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:26.739+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:26.740+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:27.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:27.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:27.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:27.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:27.312+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:27.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:27.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:27.271+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:27.271+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:27.753+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:27.753+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:27.753+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:27.754+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:28.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:28.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:28.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:28.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:28.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:28.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:28.274+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:28.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:28.275+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:28.706+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:28.706+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:28.706+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:28.707+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:29.669 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:29.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:29.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:29.302+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:29.302+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:29.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:29.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:29.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:29.670 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:29.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:29.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:29.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:29.669+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:29.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:29.670+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:30.679 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:30.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:30.679 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:30.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:30.679 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:30.299+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:30.679 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:30.300+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:30.679 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:30.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:30.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:30.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:30.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:30.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:30.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:30.283+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:30.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:30.767+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28699 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:30.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:30.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:30.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:30.678+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:30.679+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:31.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:31.319+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:31.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:31.319+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:31.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:31.320+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:31.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:31.320+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:31.566 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:31.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:31.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:31.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:31.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:31.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:31.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:31.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:31.719+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:31.719+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:32.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:32.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:32.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:32.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:32.362+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:32.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:32.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:32.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:32.283+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:32.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:32.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:32.751+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:32.751+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:33.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:33.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:33.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:33.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:33.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:33.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:33.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:33.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:33.763+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:33.763+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:34.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:34.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:34.315+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:34.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:34.316+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:34.341+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:34.341+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:34.341+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:34.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:34.341+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:35.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:34.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:35.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:34.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:35.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:34.731+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:35.053 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:34.732+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:35.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:35.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:35.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:35.352+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:35.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:35.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:35.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:35.370+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:36.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:35.771+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28787 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:35.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:36.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:35.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:36.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:35.723+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:36.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:35.724+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:36.415+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:36.415+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:36.415+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:36.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:36.415+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:36.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:36.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:36.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:36.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:36.348+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:36.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:36.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:36.772+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:36.772+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:37.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:37.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:37.362+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:37.363+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:37.370+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:37.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:37.371+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:37.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:37.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:37.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:38.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:37.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:38.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:38.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:38.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:38.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:38.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:38.336+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:38.336+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:38.336+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:38.337+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:38.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:38.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:38.782+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:39.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:38.782+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:39.408+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:39.408+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:39.408+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:39.409+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:39.382+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:39.382+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:39.382+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:39.382+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:40.092 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:39.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:40.093 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:39.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:40.093 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:39.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:40.093 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:39.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:40.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:40.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:40.427+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:40.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:40.428+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:40.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:40.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:40.435+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:40.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:40.435+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:41.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:40.774+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28877 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:40.853+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:41.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:40.853+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:40.853+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:41.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:40.854+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:41.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:41.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:41.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:41.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:41.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:41.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:41.381+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:41.382+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:41.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:41.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:41.810+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:42.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:41.811+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:42.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:42.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:42.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:42.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:42.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:42.808 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:42.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:42.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:42.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:42.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:42.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:42.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:42.344+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:42.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:42.345+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:42.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:42.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:43.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:42.786+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:43.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:42.786+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:43.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:43.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:43.425+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:43.426+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:43.345+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:43.345+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:43.345+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:43.346+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:43.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:43.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:43.759+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:43.761+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:44.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:44.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:44.390+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:44.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:44.390+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:44.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:44.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:44.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:44.463+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:45.133 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:44.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:44.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:44.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:45.134 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:44.738+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:45.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:45.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:45.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:45.426+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:45.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:45.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:45.482+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:45.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:45.483+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:46.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:45.777+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 28964 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:45.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:45.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:45.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:45.756+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:46.738 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:46.455+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:46.455+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:46.455+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:46.456+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:46.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:46.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:46.468+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:46.739 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:46.469+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:46.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:46.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:47.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:46.737+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:47.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:46.738+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:47.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:47.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:47.438+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:47.438+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:47.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:47.433+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:47.433+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:47.434+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:47.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:47.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:47.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:47.729+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:48.694 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:48.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:48.694 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:48.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:48.694 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:48.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:48.694 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:48.425+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:48.694 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:48.409+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:48.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:48.409+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:48.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:48.409+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:48.695 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:48.409+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:48.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:48.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:48.693+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:48.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:48.694+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:49.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:49.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:49.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:49.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:49.379+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:49.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:49.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:49.682 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:49.404+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:49.683 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:49.405+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:49.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:49.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:49.681+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:49.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:49.681+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:50.383+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:50.383+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:50.383+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:50.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:50.384+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:50.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:50.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:50.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:50.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:51.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:50.781+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29052 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:50.703+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:50.703+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:50.703+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:50.704+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:51.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:51.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:51.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:51.348+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:51.348+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:51.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:51.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:51.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:51.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:51.433+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:51.743+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:51.743+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:51.743+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:51.744+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:52.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:52.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:52.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:52.325+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:52.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:52.325+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:52.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:52.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:52.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:52.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:52.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:52.475+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:52.704 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:52.476+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:53.070 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:52.762+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:53.070 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:52.762+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:53.070 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:52.762+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:53.070 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:52.763+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:53.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:53.316+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:53.316+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:53.316+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:53.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:53.317+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:53.770 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:53.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:53.771 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:53.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:53.771 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:53.451+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:53.771 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:53.451+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:53.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:53.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:54.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:53.770+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:54.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:53.770+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:54.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:54.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:54.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:54.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:54.453+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:54.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:54.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:54.362+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:54.363+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:54.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:54.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:55.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:54.728+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:55.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:54.728+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:55.466 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:55.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:55.466 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:55.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:55.466 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:55.386+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:55.467 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:55.386+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:55.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:55.465+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:55.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:55.465+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:55.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:55.465+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:55.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:55.466+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:13:55.785+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29140 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:13:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:55.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:55.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:55.761+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:55.762+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:56.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:56.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:56.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:56.424+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:56.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:56.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:56.426+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:56.426+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:56.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:56.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:56.748+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:56.749+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:57.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:57.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:57.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:57.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:57.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:57.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:57.389+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:57.389+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:57.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:57.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:57.767+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:57.768+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:58.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:58.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:58.413+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:58.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:58.413+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:58.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:58.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:58.417+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:58.418+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:58.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:58.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:58.756+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:13:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:58.757+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:59.459+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:59.459+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:59.459+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:13:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:13:59.459+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:13:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:59.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:59.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:59.385+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:13:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:13:59.386+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:59.744+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:59.744+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:59.744+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:00.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:13:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:13:59.745+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:00.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:00.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:00.506 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:00.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:00.507 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:00.399+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:00.507 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:00.400+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:00.771 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:00.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:00.772 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:00.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:00.772 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:00.505+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:00.772 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:00.506+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:00.788+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29230 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:00.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:00.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:00.771+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:00.771+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:01.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:01.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:01.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:01.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:01.519+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:01.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:01.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:01.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:01.422+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:01.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:01.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:02.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:01.724+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:02.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:01.725+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:02.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:02.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:02.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:02.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:02.493+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:02.470+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:02.470+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:02.470+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:02.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:02.471+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:02.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:02.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:02.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:02.756+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:03.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:03.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:03.746+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:03.851 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:03.747+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:03.543+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:03.543+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:03.543+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:03.544+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:03.507+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:03.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:03.507+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:03.853 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:03.507+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:03.853 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:03.507+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:04.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:04.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:04.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:04.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:04.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:04.520+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:04.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:04.520+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:04.755 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:04.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:04.756 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:04.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:04.756 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:04.499+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:04.756 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:04.500+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:04.754+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:04.754+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:04.754+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:04.755+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:05.556 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:05.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:05.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:05.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:05.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:05.510+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:05.557 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:05.510+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:05.816 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:05.792+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29318 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:05.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:05.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:05.710+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:05.710+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:05.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:05.817 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:05.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:05.818 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:05.556+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:05.818 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:05.556+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:06.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:06.712+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:06.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:06.712+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:06.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:06.712+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:06.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:06.713+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:06.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:06.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:06.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:06.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:06.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:06.577+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:06.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:06.578+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:06.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:06.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:06.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:06.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:06.954 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:06.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:06.955 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:06.491+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:07.746 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:07.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:07.746 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:07.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:07.746 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:07.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:07.747 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:07.471+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:07.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:07.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:07.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:07.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:07.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:07.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:07.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:07.554+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:07.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:07.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:07.745+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:07.746+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:08.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:08.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:08.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:08.422+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:08.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:08.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:08.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:08.538+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:08.717+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:08.717+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:08.717+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:09.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:08.718+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:09.756 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:09.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:09.756 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:09.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:09.756 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:09.517+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:09.756 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:09.518+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:09.757 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:09.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:09.757 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:09.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:09.757 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:09.470+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:09.757 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:09.471+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:09.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:09.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:09.755+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:09.756+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:10.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:10.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.796 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:10.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.797 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:10.725+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:10.797 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:10.725+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:10.797 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:10.476+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:10.797 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:10.476+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:10.797 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:10.476+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:10.798 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:10.476+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:10.798 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:10.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:10.798 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:10.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:10.798 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:10.480+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:10.798 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:10.481+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:11.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:10.795+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29408 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:11.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:11.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:11.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:11.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:11.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:11.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:11.742 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:11.491+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:11.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:11.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:11.740+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:11.741+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:11.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:11.743 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:11.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:11.744 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:11.491+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:11.744 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:11.492+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:12.469+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:12.469+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:12.469+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:12.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:12.469+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:12.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:12.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:12.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:12.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:12.447+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:12.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:12.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:12.788+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:12.789+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:13.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:13.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:13.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:13.463+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:13.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:13.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:13.422+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:13.422+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:13.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:13.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:13.775+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:13.776+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:14.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:14.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:14.421+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:14.421+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:14.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:14.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:14.466+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:14.467+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:14.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:14.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:14.780+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:14.780+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:15.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:15.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:15.411+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:15.411+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:15.456+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:15.456+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:15.456+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:15.457+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:15.799+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29496 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:15.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:15.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:15.800+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:15.803+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:16.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:16.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:16.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:16.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:16.455+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:16.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:16.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:16.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:16.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:16.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:16.364+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:16.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:16.365+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:16.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:16.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:16.809+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:17.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:16.810+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:17.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:17.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:17.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:17.434+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:17.435+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:17.398+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:17.398+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:17.398+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:17.398+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:17.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:17.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:17.852+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:18.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:17.853+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:18.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:18.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:18.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:18.450+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:18.434+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:18.434+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:18.434+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:18.434+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:18.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:18.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:18.860+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:19.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:18.862+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:19.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:19.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:19.446+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:19.447+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:19.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:19.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:19.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:19.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:19.481+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:19.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:19.481+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:19.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:19.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:19.881+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:20.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:19.882+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:20.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:20.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:20.448+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:20.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:20.449+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:20.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:20.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:20.447+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:20.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:20.448+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:20.802+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29587 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:20.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:20.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:20.925+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:20.925+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:21.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:21.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:21.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:21.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:21.462+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:21.443+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:21.443+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:21.443+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:21.444+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:21.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:21.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:21.956+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:21.958+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:22.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:22.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:22.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:22.498+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:22.498+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:22.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:22.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:22.944 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:22.490+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:22.945 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:22.491+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:22.943+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:22.943+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:22.943+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:22.943+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:23.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:23.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:23.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:23.515+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:23.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:23.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:23.503+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:23.504+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:23.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:23.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:23.992+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:23.993+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:24.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:24.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:24.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:24.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:24.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:24.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:24.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:24.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:24.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:24.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:24.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:24.511+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:24.995+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:24.996+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:24.996+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:24.996+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:25.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:25.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:25.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:25.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:25.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:25.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:25.807 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:25.523+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:25.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:25.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:25.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:25.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:25.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:25.473+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:53.718707+0000 (oldest deadline 2024-03-24T11:04:13.718707+0000) 2024-03-24T11:14:25.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:25.474+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:26.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:25.806+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29675 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:25.996+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:25.996+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:26.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:25.996+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:26.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:25.997+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:26.486+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:26.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:26.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:26.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:26.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:26.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:26.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:26.473+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:27.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:27.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:27.011+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:27.012+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:27.479+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:27.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:27.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:27.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:27.454+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:28.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:28.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:28.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:28.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:28.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:28.047+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:28.441 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:28.048+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:28.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:28.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:28.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:28.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:28.440+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:29.432 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:29.063+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:29.432 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:29.063+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:29.432 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:29.063+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:54.408139+0000 (oldest deadline 2024-03-24T11:04:14.408139+0000) 2024-03-24T11:14:29.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:29.063+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:29.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:29.481+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:29.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:29.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:29.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:29.432+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:30.065+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:30.810 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:30.524+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:30.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:30.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:30.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:30.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:30.810 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:30.450+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:30.811 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:30.451+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:31.110 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:30.808+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29763 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:31.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:31.108+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:31.568+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:31.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:31.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:31.432+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:31.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:31.432+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:32.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:32.155+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:32.384+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:32.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:32.384+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:32.384+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:32.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:32.385+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:32.583+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:33.410 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:33.188+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:33.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:33.558+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:33.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:33.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:33.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:33.409+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:33.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:33.410+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:34.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:34.190+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:34.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:34.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:34.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:34.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:34.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:34.423+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:34.566+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:35.173+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:35.813 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:35.610+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:35.814 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:35.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:35.814 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:35.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:35.814 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:35.469+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:03:57.983809+0000 (oldest deadline 2024-03-24T11:04:17.983809+0000) 2024-03-24T11:14:35.814 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:35.469+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:36.156 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:35.812+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29850 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:36.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:36.154+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:36.608+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:36.497+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:37.449 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:37.179+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:37.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:37.635+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:37.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:37.448+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:38.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:38.192+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:38.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:38.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:38.589+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:39.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:39.175+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:39.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:39.622+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:40.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:40.143+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:40.337+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:40.828 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:40.816+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 29941 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:40.829 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:40.606+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:41.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:41.138+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:41.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:41.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:41.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:42.138+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:42.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:42.320+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:42.620+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:43.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:43.094+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:43.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:43.286+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:43.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:43.594+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:44.045+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:44.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:44.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:44.590+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:45.298 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:45.037+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:45.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:45.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:45.844 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:45.820+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30029 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:45.845 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:45.588+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:46.006+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:46.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:46.270+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:46.563+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:47.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:47.043+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:47.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:47.259+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:47.532+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:48.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:48.012+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:48.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:48.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:48.530+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:49.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:48.966+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:49.337+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:49.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:49.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:49.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:49.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:49.543+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:50.211 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:49.951+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:50.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:50.334+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:50.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:50.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:50.825 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:50.825 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:50.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:51.202 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:50.823+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30119 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:50.956+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:51.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:51.315+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:51.924 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:51.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:51.924 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:51.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:51.925 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:51.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:51.925 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:51.589+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:51.923+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:52.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:52.342+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:52.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:52.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:52.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:52.604+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:53.332 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:52.972+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:53.606 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:53.331+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:53.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:53.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:53.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:53.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:53.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:53.605+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:53.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:53.606+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:54.322 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:53.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:54.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:53.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:54.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:53.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:54.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:53.981+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:54.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:54.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:54.645+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:55.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:54.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:55.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:54.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:55.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:54.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:55.323 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:54.970+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:55.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:55.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:55.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:14:55.827+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30207 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:14:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:55.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:55.630+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:56.372 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:55.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:56.373 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:55.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:56.373 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:55.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:56.373 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:55.954+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:56.673 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:56.372+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:56.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:56.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:56.673+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:56.673+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:57.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:57.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:57.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:57.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:57.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:57.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:57.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:57.002+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:57.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:57.422+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:57.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:57.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:57.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:57.680+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:58.421 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:58.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:58.421 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:58.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:58.421 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:58.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:58.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:58.025+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:58.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:59.041 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:58.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.041 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:58.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.041 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:58.708+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.041 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:58.708+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:59.424 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:59.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:59.424 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:59.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:59.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:59.040+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:14:59.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:14:59.040+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:59.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:14:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:59.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:14:59.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:59.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:14:59.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:14:59.424+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:14:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:59.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:59.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:59.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:14:59.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:14:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:14:59.672+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:00.389 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:00.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:00.390 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:00.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:00.390 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:00.077+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:00.390 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:00.078+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:00.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:00.389+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:00.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:00.389+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:00.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:00.389+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:00.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:00.390+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:00.831+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30297 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:00.702+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:00.702+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:00.702+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:00.702+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:01.341 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:01.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:01.342 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:01.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:01.342 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:01.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:01.342 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:01.066+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:01.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:01.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:01.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:01.341+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:01.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:01.341+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:01.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:01.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:01.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:01.661+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:02.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:02.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:02.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:02.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:02.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:02.054+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:02.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:02.055+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:02.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:02.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:02.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:02.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:02.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:02.334+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:02.691 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:02.335+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:02.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:02.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:02.690+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:02.690+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:03.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:03.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:03.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:03.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:03.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:03.041+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:03.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:03.042+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:03.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:03.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:03.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:03.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:03.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:03.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:03.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:03.338+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:03.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:03.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:03.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:03.666+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:04.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:04.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:04.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:04.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:04.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:04.081+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:04.350 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:04.082+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:04.626 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:04.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:04.707 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:04.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:04.707 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:04.349+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:04.707 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:04.350+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:04.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:04.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:04.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:04.626+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:05.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:05.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:05.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:05.089+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:05.090+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:05.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:05.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:05.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:05.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:05.333+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:05.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:05.835+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30385 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:05.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:05.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:05.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:05.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:06.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:06.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:06.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:06.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:06.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:06.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:06.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:06.128+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:06.673 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:06.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:06.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:06.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:06.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:06.316+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:06.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:06.317+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:06.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:06.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:06.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:06.673+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:07.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:07.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:07.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:07.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:07.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:07.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:07.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:07.126+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:07.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:07.127+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:07.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:07.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:07.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:07.679+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:08.407 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:08.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:08.407 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:08.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:08.407 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:08.134+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:08.407 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:08.134+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:08.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:08.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:08.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:08.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:08.652+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:08.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:08.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:08.406+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:08.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:08.407+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:09.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:09.092+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:09.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:09.092+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:09.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:09.092+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:09.450 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:09.092+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:09.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:09.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:09.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:09.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:09.449+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:09.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:09.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:09.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:09.664+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:10.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:10.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:10.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:10.087+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:10.088+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:10.807 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:10.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:10.807 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:10.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:10.683+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:10.683+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:10.472+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:10.472+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:10.472+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:10.808 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:10.473+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:11.135 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:10.839+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30475 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:11.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:11.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:11.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:11.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:11.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:11.136+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:11.732 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:11.478+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:11.732 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:11.478+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:11.732 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:11.478+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:11.732 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:11.479+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:12.124 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:11.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:12.125 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:11.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:12.125 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:11.731+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:12.125 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:11.732+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:12.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:12.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:12.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:12.123+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:12.124+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:12.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:12.474+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:12.762 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:12.474+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:12.763 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:12.474+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:12.763 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:12.475+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:13.117 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:12.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:13.117 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:12.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:13.117 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:12.762+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:13.117 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:12.762+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:13.439 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:13.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:13.439 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:13.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:13.439 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:13.116+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:13.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:13.117+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:13.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:13.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:13.438+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:13.439+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:14.127 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:13.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:13.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:13.741+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.128 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:13.741+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:14.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:14.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:14.127+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:14.128+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:14.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:14.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:14.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:14.479+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:14.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:14.480+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:14.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:14.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:14.723+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:14.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:14.724+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:15.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:15.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:15.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:15.171+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:15.843 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:15.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:15.844 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:15.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:15.844 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:15.496+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:15.844 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:15.497+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:15.844 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:15.705+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:15.844 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:15.705+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:15.845 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:15.705+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:15.845 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:15.706+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:15.843+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30561 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:16.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:16.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:16.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:16.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:16.463 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:16.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:16.464 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:16.462+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:16.464 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:16.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:16.464 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:16.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:16.464 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:16.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:16.464 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:16.213+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:16.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:16.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:16.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:16.685+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:17.510 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:17.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:17.510 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:17.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:17.510 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:17.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:17.511 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:17.211+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:17.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:17.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:17.695+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:17.695+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:17.509+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:17.509+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:17.509+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:17.510+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:18.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:18.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:18.170+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:18.170+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:18.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:18.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:18.556+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:18.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:18.557+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:18.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:18.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:18.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:18.661+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:19.517 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:19.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:19.517 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:19.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:19.517 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:19.217+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:19.517 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:19.218+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:19.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:19.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:19.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:19.649+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:19.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:19.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:19.516+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:19.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:19.516+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:20.493 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:20.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:20.493 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:20.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:20.493 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:20.230+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:20.494 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:20.231+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:20.847 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:20.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:20.847 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:20.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:20.847 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:20.492+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:20.847 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:20.493+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:20.848 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:20.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:20.848 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:20.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:20.848 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:20.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:20.848 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:20.603+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:20.846+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30651 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:21.508 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:21.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:21.509 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:21.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:21.509 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:21.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:21.509 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:21.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:21.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:21.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:21.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:21.563+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:21.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:21.508+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:21.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:21.508+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:21.953 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:21.508+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:21.953 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:21.508+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:22.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:22.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:22.245+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:22.246+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:22.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:22.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:22.477+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:22.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:22.478+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:22.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:22.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:22.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:22.584+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:23.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:23.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:23.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:23.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:23.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:23.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:23.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:23.243+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:23.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:23.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:23.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:23.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:23.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:23.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:23.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:23.485+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:23.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:23.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:23.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:23.594+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:24.488 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:24.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:24.489 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:24.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:24.489 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:24.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:24.489 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:24.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:24.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:24.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:24.488+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:24.488+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:24.584+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:24.584+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:24.584+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:24.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:24.585+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:25.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:25.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:25.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:25.194+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:25.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:25.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:25.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:25.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:25.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:25.580+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:25.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:25.580+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:25.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:25.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:25.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:25.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:25.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:25.518+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:25.852 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:25.518+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:26.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:25.850+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30740 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:26.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:26.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:26.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:26.234+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:26.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:26.235+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:26.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:26.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:26.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:26.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:26.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:26.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:26.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:26.477+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:26.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:26.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:26.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:26.573+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:27.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:27.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:27.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:27.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:27.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:27.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:27.200+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:27.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:27.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:27.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:27.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:27.474+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:27.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:27.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:27.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:27.599+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:28.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:28.175+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:28.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:28.175+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:28.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:28.175+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:28.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:28.176+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:28.441+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:28.441+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:28.441+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:28.442+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:28.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:28.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:28.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:28.632+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:29.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:29.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:29.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:29.202+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:29.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:29.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:29.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:29.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:29.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:29.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:29.429+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:29.659+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:29.659+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:29.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:29.659+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:29.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:29.659+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:30.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:30.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:30.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:30.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:30.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:30.179+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:30.444 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:30.180+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:30.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:30.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:30.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:30.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:30.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:30.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:30.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:30.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:30.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:30.646+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:31.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:30.854+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30829 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:31.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:31.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:31.221+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:31.222+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:31.453+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:31.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:31.453+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:31.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:31.453+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:31.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:31.454+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:31.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:31.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:31.666+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:31.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:31.666+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:32.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:32.269+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:32.269+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:32.269+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:32.270+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:32.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:32.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:32.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:32.422+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:32.422+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:32.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:32.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:32.699+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:32.700+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:33.698 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:33.285+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:33.285+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:33.285+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:33.286+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:33.433+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:33.433+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:33.699 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:33.433+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:33.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:33.434+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:33.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:33.698+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:33.698+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:33.698+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:34.699 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:34.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:34.699 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:34.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:34.262+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:34.262+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:34.457+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:34.457+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:34.457+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:34.700 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:34.458+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:34.698+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:34.698+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:34.698+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:34.699+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:35.486 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:35.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:35.486 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:35.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:35.486 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:35.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:35.486 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:35.234+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:35.859 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:35.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:35.859 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:35.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:35.859 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:35.485+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:35.859 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:35.486+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:35.860 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:35.654+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:35.860 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:35.654+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:35.860 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:35.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:35.860 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:35.655+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:36.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:35.858+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 30918 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:36.642 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:36.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:36.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:36.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:36.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:36.436+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:36.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:36.436+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:36.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:36.213+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:36.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:36.213+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:36.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:36.213+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:36.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:36.214+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:36.641+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:36.641+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:36.641+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:36.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:37.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:37.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:37.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:37.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:37.428+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:37.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:37.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:37.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:37.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:37.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:37.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:37.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:37.199+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:37.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:37.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:37.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:37.626+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:38.658 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:38.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:38.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:38.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:38.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:38.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:38.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:38.202+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:38.659 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:38.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:38.659 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:38.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:38.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:38.431+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:38.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:38.431+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:38.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:38.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:38.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:38.658+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:39.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:39.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:39.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:39.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:39.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:39.236+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:39.666 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:39.237+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:39.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:39.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:39.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:39.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:39.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:39.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:39.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:39.384+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:39.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:39.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:39.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:39.664+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:40.589 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:40.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:40.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:40.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:40.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:40.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:40.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:40.237+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:40.590 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:40.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:40.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:40.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:40.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:40.405+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:40.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:40.405+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:40.863 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:40.638+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:40.863 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:40.638+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:40.863 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:40.638+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:40.864 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:40.638+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:41.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:40.861+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31007 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:41.642 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:41.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:41.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:41.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:41.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:41.215+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:41.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:41.215+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:41.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:41.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:41.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:41.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:41.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:41.397+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:41.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:41.398+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:41.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:41.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:41.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:41.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:41.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:42.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:42.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:42.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:42.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:42.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:42.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:42.645 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:42.248+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:42.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:42.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:42.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:42.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:42.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:42.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:42.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:42.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:42.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:42.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:42.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:42.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:43.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:43.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:43.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:43.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:43.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:43.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:43.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:43.388+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:43.661 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:43.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:43.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:43.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:43.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:43.265+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:43.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:43.266+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:43.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:43.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:43.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:43.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:43.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:43.660+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:44.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:44.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:44.284+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:44.614 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:44.284+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:44.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:44.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:44.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:44.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:44.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:44.388+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:44.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:44.388+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:44.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:44.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:44.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:44.613+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:45.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:45.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:45.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:45.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:45.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:45.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:45.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:45.322+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:45.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:45.322+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:45.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:45.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:45.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:45.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:45.418+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:45.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:45.419+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:46.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:45.866+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31096 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:46.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:46.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:46.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:46.543+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:46.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:46.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:46.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:46.345+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:46.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:46.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:46.398+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:46.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:46.399+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:47.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:47.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:47.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:47.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:47.553+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:47.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:47.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:47.365+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:47.366+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:47.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:47.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:47.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:47.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:47.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:47.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:47.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:47.305+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:48.562 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:48.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:48.562 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:48.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:48.562 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:48.300+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:48.563 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:48.301+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:48.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:48.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:48.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:48.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:48.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:48.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:48.563 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:48.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:48.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:48.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:48.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:48.561+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:49.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:49.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:49.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:49.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:49.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:49.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:49.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:49.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:49.269+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:49.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:49.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:49.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:49.546+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:50.551 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:50.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:50.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:50.552 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:50.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:50.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:50.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:50.552 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:50.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:50.553 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:50.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:50.553 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:50.256+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:50.871 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:50.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:50.871 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:50.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:50.871 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:50.551+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:50.871 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:50.551+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:51.137 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:50.870+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31185 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:51.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:51.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:51.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:51.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:51.348+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:51.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:51.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:51.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:51.211+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:51.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:51.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:51.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:51.598+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:51.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:51.599+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:52.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:52.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:52.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:52.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:52.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:52.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:52.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:52.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:52.256+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:52.585 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:52.257+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:52.877 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:52.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:52.878 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:52.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:52.878 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:52.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:52.878 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:52.583+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:53.597 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:53.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:53.597 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:53.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:53.598 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:53.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:53.598 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:53.305+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:53.598 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:53.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:53.598 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:53.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:53.598 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:53.273+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:53.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:53.273+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:53.596+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:53.596+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:53.596+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:53.597+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:54.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:54.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:54.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:54.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:54.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:54.324+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:54.324+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:54.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:54.324+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:54.647 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:54.325+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:54.644+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:54.645+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:55.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:55.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:55.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:55.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:55.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:55.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:55.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:55.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:55.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:55.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:55.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:55.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:55.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:55.341+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:55.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:55.342+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:55.886 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:15:55.873+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31274 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:15:55.887 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:55.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:55.887 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:55.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:55.887 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:55.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:55.887 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:55.628+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:56.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:56.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:56.315+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:56.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:56.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:56.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:56.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:56.376+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:56.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:56.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:56.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:56.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:57.617 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:57.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:57.618 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:57.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:57.618 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:57.370+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:57.618 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:57.371+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:57.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:57.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:57.619 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:57.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:57.619 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:57.285+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:57.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:57.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:57.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:57.617+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:58.658 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:58.329+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:58.329+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:58.329+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:58.330+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:58.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:58.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:58.659 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:58.303+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:58.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:58.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:58.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:58.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:58.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:58.658+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:59.670 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:59.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:59.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:59.352+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:15:59.353+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:59.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:59.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:59.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:15:59.671 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:15:59.311+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:15:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:59.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:59.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:59.669+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:15:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:15:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:15:59.670+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:00.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:00.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:00.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:00.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:00.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:00.330+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:00.687 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:00.331+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:00.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:00.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:00.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:00.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:00.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:00.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:00.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:00.301+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:00.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:00.877+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31364 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:00.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:00.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:00.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:00.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:00.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:00.686+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:01.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:01.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:01.299+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:01.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:01.299+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:01.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:01.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:01.279+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:01.280+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:02.093 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:02.094 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:02.094 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:01.724+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:02.094 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:01.725+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:02.263+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:02.263+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:02.263+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:02.265+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:02.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:02.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:02.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:02.306+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:02.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:02.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:02.742+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:03.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:02.743+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:03.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:03.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:03.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:03.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:03.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:03.253+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:03.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:03.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:03.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:03.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:03.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:04.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:03.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:03.764+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:04.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:03.764+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:04.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:04.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:04.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:04.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:04.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:04.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:04.247+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:04.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:04.247+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:04.754+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:04.754+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:05.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:04.754+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:05.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:04.754+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:05.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:05.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:05.345+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:05.345+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:05.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:05.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:05.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:05.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:05.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:06.181 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:05.881+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31452 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:06.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:05.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:06.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:05.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:06.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:05.770+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:06.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:05.770+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:06.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:06.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:06.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:06.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:06.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:06.181+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:06.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:06.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:06.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:06.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:06.371+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:07.165 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:06.801+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:07.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:06.801+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:07.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:06.801+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:07.166 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:06.802+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:07.419 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:07.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:07.420 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:07.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:07.420 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:07.165+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:07.420 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:07.165+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:07.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:07.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:07.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:07.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:07.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:07.419+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:08.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:07.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:08.181 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:07.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:08.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:07.763+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:08.182 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:07.763+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:08.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:08.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:08.180+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:08.181+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:08.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:08.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:08.392+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:08.392+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:08.738+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:09.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:08.738+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:08.738+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:09.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:08.738+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:09.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:09.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:09.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:09.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:09.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:09.210+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:09.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:09.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:09.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:09.400+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:09.709+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:09.709+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:09.709+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:10.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:09.710+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:10.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:10.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:10.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:10.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:10.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:10.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:10.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:10.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:10.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:10.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:10.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:10.208+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:11.009 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:10.885+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31541 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:11.009 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:10.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:11.009 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:10.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:11.009 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:10.733+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:11.010 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:10.734+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:11.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:11.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:11.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:11.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:11.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:11.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:11.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:11.169+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:11.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:11.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:11.361+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:11.362+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:12.129 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:11.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:12.129 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:11.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:12.129 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:11.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:12.130 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:11.721+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:12.396 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:12.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:12.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:12.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:12.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:12.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:12.397 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:12.129+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:12.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:12.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:12.395+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:12.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:12.396+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:13.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:12.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:13.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:12.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:13.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:12.721+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:13.099 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:12.721+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:13.355 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:13.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:13.355 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:13.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:13.355 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:13.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:13.355 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:13.099+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:13.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:13.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:13.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:13.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:14.086 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:13.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.087 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:13.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.087 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:13.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.087 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:13.708+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:14.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:14.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:14.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:14.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:14.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:14.086+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:14.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:14.086+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:14.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:14.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:14.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:14.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:14.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:14.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:14.661 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:14.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:14.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:14.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:14.660+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:14.660+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:15.429 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:15.429 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:15.429 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:15.043+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:15.429 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:15.043+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:15.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:15.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:15.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:15.636+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:15.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:15.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:15.428+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:15.429+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:15.889+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31630 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:16.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:16.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:16.042+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:16.042+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:16.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:16.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:16.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:16.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:16.687 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:16.442+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:16.688 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:16.443+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:16.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:16.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:16.686+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:16.686+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:17.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:17.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:17.064+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:17.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:17.065+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:17.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:17.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:17.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:17.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:17.659+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:17.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:17.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:17.463+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:17.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:17.464+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:18.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:18.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:18.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:18.082+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:18.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:18.083+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:18.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:18.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:18.494+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:18.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:18.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:18.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:18.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:18.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:18.636+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:19.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:19.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:19.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:19.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:19.102+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:19.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:19.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:19.462+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:19.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:19.462+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:19.624+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:19.624+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:19.624+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:19.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:20.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:20.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:20.088+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:20.089+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:20.894 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:20.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:20.894 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:20.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:20.894 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:20.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:20.895 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:20.496+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:20.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:20.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:20.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:20.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:20.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:20.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:20.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:20.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:20.893+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31720 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:21.120+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:21.120+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:21.120+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:21.121+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:21.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:21.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:21.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:21.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:21.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:21.476+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:21.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:21.477+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:21.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:21.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:21.663+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:21.664+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:22.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:22.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:22.149+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:22.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:22.149+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:22.512+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:22.512+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:22.512+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:22.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:22.513+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:22.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:22.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:22.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:22.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:22.667+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:23.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:23.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:23.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:23.145+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:23.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:23.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:23.697+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:23.698+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:23.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:23.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:23.853 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:23.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:23.853 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:23.562+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:23.853 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:23.562+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:24.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:24.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:24.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:24.097+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:24.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:24.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:24.568+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:24.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:24.569+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:24.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:24.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:24.707+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:24.708+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:25.121+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:25.121+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:25.121+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:25.121+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:25.899 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:25.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:25.901 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:25.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:25.901 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:25.604+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:25.901 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:25.604+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:25.902 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:25.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:25.902 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:25.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:25.902 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:25.684+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:25.902 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:25.684+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:26.160 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:25.897+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31806 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:26.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:26.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:26.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:26.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:26.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:26.160+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:26.423 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:26.161+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:26.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:26.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:26.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:26.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:26.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:26.635+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:26.651+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:26.651+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:26.651+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:26.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:26.652+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:27.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:27.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:27.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:27.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:27.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:27.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:27.165+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:27.676+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:27.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:27.676+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:27.676+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:27.677+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:27.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:27.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:27.662+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:27.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:27.663+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:28.634 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:28.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:28.635 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:28.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:28.635 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:28.210+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:28.635 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:28.210+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:28.703+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:28.703+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:28.703+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:28.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:28.704+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:28.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:28.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:28.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:28.634+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:29.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:29.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:29.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:29.198+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:29.198+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:29.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:29.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:29.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:29.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:29.662+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:29.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:29.742+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:29.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:29.742+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:29.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:29.742+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:29.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:29.743+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:30.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:30.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:30.191+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:30.192+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:30.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:30.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:30.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:30.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:30.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:30.725+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:30.902 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:30.725+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:30.902 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:30.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:30.903 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:30.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:30.903 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:30.648+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:30.903 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:30.648+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:31.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:30.901+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31889 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:31.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:31.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:31.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:31.146+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:31.747 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:31.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:31.748 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:31.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:31.748 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:31.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:31.748 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:31.604+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:32.146 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:31.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.146 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:31.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.146 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:31.747+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.146 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:31.747+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:32.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:32.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:32.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:32.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:32.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:32.145+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:32.454 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:32.146+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:32.720+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:32.720+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:32.720+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:32.721+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:32.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:32.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:32.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:32.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:32.653+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:33.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:33.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:33.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:33.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:33.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:33.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:33.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:33.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:33.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:33.611+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:33.726+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:33.726+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:33.726+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:33.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:33.727+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:34.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:34.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:34.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:34.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:34.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:34.208+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:34.662 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:34.209+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:34.690+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:34.690+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:34.690+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:34.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:34.691+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:34.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:34.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:34.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:34.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:34.661+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:35.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:35.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:35.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:35.178+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:35.179+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:35.917 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:35.905+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 31972 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:35.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:35.705+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:35.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:35.705+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:35.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:35.705+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:35.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:35.706+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:35.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:35.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:35.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:35.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:35.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:35.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:35.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:35.662+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:36.506 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:36.226+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:36.506 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:36.226+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:36.506 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:36.226+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:36.506 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:36.227+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:36.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:36.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:36.666+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:36.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:36.667+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:36.618+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:36.618+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:36.618+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:36.618+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:37.647 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:37.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:37.647 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:37.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:37.648 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:37.248+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:37.648 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:37.248+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:37.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:37.646+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:37.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:37.646+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:37.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:37.646+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:37.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:37.647+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:37.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:37.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:37.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:37.655+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:38.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:38.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:38.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:38.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:38.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:38.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:38.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:38.253+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:38.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:38.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:38.609+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:38.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:38.609+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:38.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:38.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:38.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:38.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:39.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:39.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:39.591 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:39.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:39.591 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:39.240+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:39.591 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:39.240+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:39.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:39.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:39.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:39.637+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:39.638+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:39.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:39.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:39.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:39.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:39.590+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:40.555 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:40.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:40.555 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:40.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:40.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:40.289+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:40.556 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:40.290+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:40.910 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:40.621+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:40.910 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:40.621+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:40.910 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:40.621+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:40.910 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:40.621+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:40.911 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:40.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:40.911 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:40.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:40.911 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:40.554+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:40.911 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:40.555+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:41.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:40.909+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32053 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:41.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:41.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:41.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:41.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:41.545 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:41.287+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:41.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:41.287+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:41.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:41.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:41.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:41.618+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:41.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:41.619+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:41.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:41.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:41.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:41.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:41.594+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:42.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:42.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:42.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:42.249+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:42.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:42.249+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:42.605+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:42.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:42.605+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:42.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:42.605+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:42.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:42.606+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:42.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:42.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:42.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:42.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:42.623+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:43.575 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:43.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:43.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:43.202+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:43.576 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:43.203+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:43.575+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:43.575+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:43.575+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:43.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:43.575+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:43.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:43.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:43.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:43.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:43.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:43.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:43.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:43.628+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:44.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:44.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:44.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:44.223+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:44.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:44.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:44.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:44.626+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:44.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:44.570+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:44.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:44.570+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:44.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:44.570+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:44.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:44.570+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:45.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:45.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:45.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:45.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:45.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:45.206+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:45.534 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:45.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:45.913 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:45.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:45.914 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:45.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:45.914 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:45.532+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:45.914 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:45.533+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:45.914 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:45.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:45.914 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:45.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:45.915 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:45.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:45.915 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:45.586+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:46.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:45.913+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32133 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:46.523 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:46.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:46.523 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:46.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:46.523 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:46.233+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:46.524 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:46.234+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:46.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:46.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:46.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:46.522+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:46.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:46.523+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:46.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:46.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:46.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:46.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:46.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:46.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:46.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:46.588+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:47.474 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:47.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:47.474 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:47.200+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:47.475 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:47.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:47.475 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:47.202+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:47.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:47.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:47.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:47.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:47.474+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:47.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:47.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:47.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:47.609+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:48.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:48.225+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:48.225+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:48.225+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:48.226+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:48.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:48.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:48.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:48.449+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:48.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:48.450+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:48.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:48.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:48.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:48.635+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:48.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:48.635+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:49.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:49.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:49.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:49.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:49.196+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:49.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:49.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:49.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:49.473+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:49.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:49.474+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:49.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:49.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:49.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:49.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:49.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:49.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:49.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:49.595+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:50.166+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:50.166+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:50.166+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:50.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:50.167+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:50.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:50.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:50.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:50.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:50.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:50.460+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:50.918 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:50.461+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:50.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:50.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:50.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:50.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:50.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:50.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:51.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:50.916+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32216 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:51.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:51.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:51.135+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:51.203 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:51.135+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:51.567 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:51.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:51.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:51.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:51.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:51.471+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:51.568 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:51.471+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:51.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:51.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:51.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:51.567+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:52.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:52.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:52.132+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:52.133+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:52.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:52.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:52.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:52.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:52.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:52.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:52.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:52.454+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:52.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:52.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:52.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:52.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:52.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:52.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:52.599+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:53.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:53.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:53.164+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:53.164+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:53.850 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:53.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:53.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:53.495+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:53.851 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:53.495+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:53.586+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:53.586+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:53.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:53.586+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:53.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:53.586+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:54.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:54.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:54.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:54.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:54.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:54.205+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:54.590 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:54.206+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:54.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:54.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:54.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:54.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:54.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:54.454+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:54.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:54.455+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:54.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:54.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:54.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:54.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:54.587+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:55.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:55.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:55.199+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:55.199+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:55.922 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:55.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:55.922 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:55.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:55.467+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:55.467+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:55.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:55.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:55.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:55.923 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:55.615+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:16:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:16:55.921+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32302 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:16:56.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:56.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:56.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:56.201+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:56.202+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:56.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:56.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:56.665 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:56.439+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:56.666 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:56.439+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:56.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:56.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:56.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:56.664+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:56.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:56.664+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:57.656 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:57.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:57.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:57.424+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:57.425+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:57.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:57.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:57.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:57.227+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:57.658 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:57.228+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:57.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:57.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:57.655+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:57.656+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:58.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:58.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:58.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:58.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:58.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:58.401+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:58.643 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:58.402+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:58.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:58.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:58.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:58.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:58.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:58.219+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:58.644 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:58.220+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:58.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:58.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:58.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:58.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:58.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:58.643+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:59.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:59.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:59.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:59.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:59.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:59.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:16:59.693 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:16:59.357+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:59.694 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:59.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:59.694 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:59.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:59.694 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:59.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:16:59.694 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:16:59.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:16:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:59.692+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:59.692+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:59.692+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:16:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:16:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:16:59.693+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:00.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:00.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:00.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:00.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:00.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:00.369+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:00.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:00.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:00.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:00.669 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:00.223+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:00.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:00.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:00.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:00.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:00.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:00.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:00.926 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:00.668+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:00.925+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32392 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:01.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:01.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:01.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:01.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:01.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:01.211+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:01.681 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:01.211+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:01.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:01.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:01.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:01.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:01.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:01.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:01.681 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:01.374+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:01.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:01.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:01.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:01.679+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:01.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:01.680+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:02.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:02.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:02.188+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:02.189+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:02.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:02.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:02.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:02.326+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:03.185 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:02.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.186 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:02.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.186 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:02.719+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.186 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:02.719+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:03.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:03.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:03.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:03.184+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:03.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:03.185+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:03.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:03.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:03.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:03.301+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:03.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:03.302+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:03.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:03.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:03.675+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:03.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:03.675+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:04.682 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:04.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:04.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:04.222+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:04.222+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:04.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:04.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:04.683 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:04.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:04.684 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:04.268+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:04.681+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:04.681+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:04.681+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:04.682+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:05.666 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:05.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:05.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:05.204+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:05.205+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:05.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:05.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:05.667 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:05.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:05.668 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:05.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:05.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:05.665+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:05.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:05.665+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:05.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:05.665+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:05.931 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:05.666+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:06.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:05.929+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32481 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:06.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:06.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:06.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:06.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:06.268+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:06.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:06.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:06.207+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:06.207+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:06.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:06.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.201 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:06.706+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.202 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:06.706+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:07.663 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:07.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:07.663 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:07.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:07.663 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:07.237+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:07.663 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:07.238+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:07.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:07.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:07.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:07.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:07.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:07.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:07.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:07.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:07.661+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:07.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:07.662+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:07.662+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:08.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:08.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:08.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:08.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:08.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:08.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:08.221+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:08.222+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:08.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:08.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:08.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:08.630+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:09.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:09.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:09.185+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:09.186+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:09.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:09.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:09.234+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:09.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:09.235+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:09.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:09.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:09.621+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:09.622+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:10.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:10.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:10.282+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:10.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:10.283+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:10.168+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:10.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:10.169+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:10.934 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:10.667+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:10.934 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:10.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:10.935 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:10.668+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:10.935 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:10.668+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:11.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:10.933+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32570 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:11.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:11.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:11.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:11.142+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:11.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:11.143+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:11.624 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:11.625 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:11.625 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:11.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:11.625 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:11.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:11.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:11.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:11.623+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:11.624+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:12.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:12.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:12.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:12.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:12.315+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:12.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:12.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:12.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:12.123+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:12.941 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:12.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:12.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:12.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:12.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:12.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:12.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:12.613+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:13.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:13.151+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:13.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:13.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:13.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:13.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:13.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:13.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:13.600 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:13.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:13.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:13.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:13.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:13.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:13.852 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:13.600+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:14.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:14.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:14.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:14.309+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:14.310+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:14.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:14.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:14.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:14.153+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:14.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:14.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:14.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:14.590+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:15.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:15.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:15.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:15.139+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:15.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:15.140+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:15.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:15.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:15.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:15.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:15.286+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:15.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:15.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:15.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:15.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:15.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:15.549+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:15.937+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32659 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:16.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:16.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:16.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:16.099+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:16.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:16.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:16.573 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:16.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:16.573 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:16.311+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:16.573 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:16.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:16.828 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:16.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:16.828 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:16.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:16.828 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:16.571+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:16.829 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:16.572+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:17.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:17.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:17.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:17.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:17.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:17.104+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:17.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:17.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:17.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:17.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:17.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:17.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:17.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:17.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:17.578+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:18.299 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:18.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:18.299 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:18.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:18.299 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:18.122+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:18.300 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:18.123+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:18.592 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:18.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:18.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:18.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:18.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:18.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:18.593 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:18.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:18.591+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:18.591+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:18.591+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:18.592+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:19.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:19.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:19.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:19.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:19.084+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:19.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:19.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:19.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:19.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:19.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:19.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:19.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:19.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:19.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:19.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:19.611+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:19.612+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:20.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:20.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:20.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:20.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:20.125+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:20.261+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:20.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:20.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:20.942 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:20.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:20.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:20.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:20.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:20.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:20.943 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:20.573+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:21.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:20.941+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32749 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:21.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:21.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:21.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:21.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:21.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:21.097+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:21.544 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:21.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:21.544 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:21.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:21.544 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:21.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:21.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:21.286+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:21.867 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:21.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:21.867 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:21.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:21.867 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:21.543+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:21.867 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:21.544+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:22.157 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:22.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:22.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:22.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:22.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:22.115+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:22.158 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:22.116+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:22.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:22.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:22.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:22.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:22.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:22.244+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:22.454 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:22.245+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:22.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:22.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:22.529+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:22.530+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:23.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:23.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:23.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:23.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:23.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:23.084+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:23.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:23.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:23.247+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:23.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:23.247+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:23.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:23.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:23.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:23.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:23.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:23.502+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:24.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:24.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:24.093+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:24.094+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:24.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:24.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:24.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:24.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:24.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:24.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:24.525 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:24.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:24.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:24.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:24.524+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:24.524+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:25.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:25.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:25.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:25.128+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:25.129+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:25.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:25.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:25.198+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:25.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:25.198+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:25.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:25.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:25.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:25.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:25.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:25.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:25.947 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:25.527+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:26.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:25.946+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32838 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:26.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:26.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:26.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:26.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:26.096+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:26.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:26.097+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:26.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:26.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:26.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:26.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:26.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:26.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:26.502 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:26.248+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:26.905 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:26.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:26.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:26.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:26.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:26.501+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:26.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:26.501+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:27.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:27.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:27.083+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:27.084+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:27.461 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:27.461 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:27.461 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:27.204+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:27.461 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:27.205+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:27.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:27.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:27.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:27.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:27.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:27.460+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:28.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:28.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:28.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:28.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:28.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:28.105+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:28.433 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:28.106+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:28.433 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:28.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:28.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:28.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:28.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:28.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:28.434 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:28.204+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:28.432+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:28.432+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:28.432+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:28.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:28.432+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:29.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:29.177+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:29.178+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:29.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:29.179+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:29.151+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:29.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:29.152+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:29.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:29.153+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:29.460+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:30.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:30.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:30.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:30.107+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:30.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:30.108+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:30.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:30.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:30.223+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:30.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:30.223+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:30.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:30.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:30.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:30.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:30.478+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:30.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:30.478+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:31.373 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:30.950+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 32927 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:31.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:31.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:31.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:31.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:31.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:31.109+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:31.374 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:31.110+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:31.375 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:31.174+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:31.375 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:31.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:31.375 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:31.175+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:31.375 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:31.175+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:31.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:31.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:31.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:31.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:31.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:32.084+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:32.084+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:32.084+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:32.085+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:32.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:32.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:32.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:32.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:32.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:32.213+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:32.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:32.213+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:32.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:32.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:32.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:32.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:32.534+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:33.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:33.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:33.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:33.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:33.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:33.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:33.253+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:33.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:33.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:33.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:33.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:33.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:33.103+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:33.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:33.103+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:33.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:33.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:33.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:33.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:33.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:33.514+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:33.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:33.514+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:34.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:34.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:34.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:34.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:34.065+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:34.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:34.065+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:34.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:34.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:34.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:34.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:34.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:34.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:34.541 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:34.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:34.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:34.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:34.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:34.541+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:35.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:35.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:35.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:35.267+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:35.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:35.268+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:35.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:35.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:35.037+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:35.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:35.038+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:35.902 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-24T11:17:35.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:17:35.907 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-24T11:17:35.908 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:17:35.923 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:17:35.934 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:17:35.934 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:17:35.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:17:35.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:35.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:35.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:35.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:35.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:35.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:35.954 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:35.565+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:35.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:17:35.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:17:36.237 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:35.954+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33016 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:36.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.017+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:36.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.017+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:36.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.017+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:36.237 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.017+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:36.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:36.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:36.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:36.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:36.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:36.236+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:36.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:36.236+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:36.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:36.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:36.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:36.594+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:36.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:36.595+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:37.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.980+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:37.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:36.981+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:37.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:37.191+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:37.192+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:37.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:37.192+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:37.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:37.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:37.562+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:37.562+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:38.215 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:37.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:38.217 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:37.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:38.218 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:37.953+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:38.218 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:37.954+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:38.571 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:38.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:38.571 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:38.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:38.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:38.215+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:38.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:38.215+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:38.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:38.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:38.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:38.571+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:39.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:17:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:39.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:39.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:39.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:39.171+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:39.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:39.172+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:39.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:38.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:39.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:38.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:39.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:38.972+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:39.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:38.973+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:39.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:39.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:39.617+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:39.618+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:40.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:40.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:40.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:40.195+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:40.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:40.196+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:40.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:40.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:40.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.001+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:40.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.002+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:40.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:40.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:40.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:40.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:40.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:40.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:40.611+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:41.225 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:40.958+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33106 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:41.225 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:41.226 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:41.226 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.959+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:41.226 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:40.960+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:41.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:41.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:41.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:41.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:41.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:41.224+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:41.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:41.225+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:41.929 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:41.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:41.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:41.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:41.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:41.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:41.930 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:41.595+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:41.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:41.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:41.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:42.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:41.929+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:42.634 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:42.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:42.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:42.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:42.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:42.274+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:42.635 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:42.275+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:42.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:42.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:42.634+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:42.634+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:43.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:42.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:43.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:42.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:43.227 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:42.962+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:43.228 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:42.963+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:43.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:43.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:43.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:43.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:43.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:43.226+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:43.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:43.227+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:43.937 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:43.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:43.937 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:43.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:43.937 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:43.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:43.938 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:43.614+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:43.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:43.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:44.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:43.936+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:43.937+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:44.633 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:44.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:44.633 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:44.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:44.633 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:44.225+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:44.633 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:44.226+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:44.922 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:44.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:44.922 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:44.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:44.922 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:44.632+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:44.922 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:44.632+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:44.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:44.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:45.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:44.921+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:45.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:44.922+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:45.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:45.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:45.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:45.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:45.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:45.205+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:45.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:45.206+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:45.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:45.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:45.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:45.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:45.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:45.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:45.946 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:45.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:46.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:45.963+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33195 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:46.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:45.945+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:45.945+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:45.945+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:46.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:45.946+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:46.579 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:46.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:46.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:46.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:46.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:46.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:46.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:46.253+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:46.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:46.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:46.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:46.579+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:46.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:46.579+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:47.303 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:46.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:47.304 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:46.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:47.304 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:46.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:47.304 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:46.983+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:47.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:47.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:47.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:47.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:47.614 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:47.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:47.615 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:47.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:47.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:47.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:47.613+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:47.614+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:48.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:17:48.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:17:48.289 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:47.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:48.290 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:47.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:48.290 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:47.970+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:48.290 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:47.970+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:48.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:48.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:48.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:48.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:48.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:48.288+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:48.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:48.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:48.928 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:48.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:48.928 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:48.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:48.929 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:48.626+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:48.929 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:48.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:48.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:48.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:48.927+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:48.928+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:49.673 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:49.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:49.673 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:49.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:49.673 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:49.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:49.674 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:49.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:49.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:49.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:49.672+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:49.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:49.673+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:49.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:49.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:49.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:49.914+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:49.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:49.914+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:50.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:50.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:50.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:50.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:50.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:50.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:50.640 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:50.273+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:50.894 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:50.639+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:50.894 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:50.639+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:50.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:50.639+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:50.895 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:50.639+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:51.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:17:51.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:50.967+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33284 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:50.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:51.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:50.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:50.893+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:51.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:50.894+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:51.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:51.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:51.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:51.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:51.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:51.272+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:51.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:51.272+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:51.920 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:51.657+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:51.920 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:51.657+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:51.921 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:51.657+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:51.921 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:51.657+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:51.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:51.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:51.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:51.920+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:52.677 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:52.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:52.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:52.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:52.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:52.275+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:52.678 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:52.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:52.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:52.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:52.922+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:52.923+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:52.677+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:52.677+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:52.677+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:52.677+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:53.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:53.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:53.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:53.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:53.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:53.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:53.690 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:53.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:53.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:53.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:53.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:53.908+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:53.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:53.908+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:53.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:53.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:53.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:53.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:53.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:53.689+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:53.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:53.689+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:54.647 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:54.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:54.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:54.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:54.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:54.277+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:54.648 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:54.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:54.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:54.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:54.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:54.646+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:54.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:54.647+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:54.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:54.858+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:54.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:54.859+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:54.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:54.859+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:54.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:54.859+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:55.631 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:55.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:55.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:55.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:55.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:55.264+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:55.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:55.265+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:55.882 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:55.883 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:55.883 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:55.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:55.883 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:55.631+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:56.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:17:55.971+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33373 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:17:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:55.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:56.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:55.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:55.882+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:56.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:55.882+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:56.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:56.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:56.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:56.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:56.586 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:56.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:56.587 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:56.314+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:56.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:56.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:56.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:56.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:56.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:56.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:56.906 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:56.586+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:57.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:56.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:56.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:56.905+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:57.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:56.905+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:57.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:57.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:57.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:57.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:57.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:57.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:57.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:57.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:57.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:57.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:57.565+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:57.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:58.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:57.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:57.895+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:58.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:57.896+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:58.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:58.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:58.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:58.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:58.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:58.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:58.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:58.564+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:58.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:58.565+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:58.915+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:59.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:58.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:58.916+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:17:59.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:58.916+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:59.579 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:59.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:59.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:59.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:59.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:59.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:17:59.580 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:17:59.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:17:59.874 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:59.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:59.874 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:59.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:59.874 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:59.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:17:59.874 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:17:59.578+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:59.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:59.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:00.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:59.869+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:00.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:17:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:17:59.870+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:00.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:00.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:00.539 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:00.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:00.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:00.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:00.540 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:00.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:00.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:00.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:00.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:00.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:00.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:00.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:00.824 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:00.538+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:01.101 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:00.975+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33463 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:01.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:00.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:01.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:00.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:01.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:00.822+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:01.101 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:00.822+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:01.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:01.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:01.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:01.490+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:01.490+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:01.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:01.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:01.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:01.373+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:01.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:01.374+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:02.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:01.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:02.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:01.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:02.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:01.798+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:02.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:01.799+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:02.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:02.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:02.447+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:02.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:02.447+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:02.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:02.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:02.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:02.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:02.368+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:03.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:03.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:03.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:02.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:02.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:02.834+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:03.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:02.835+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:03.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:03.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:03.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:03.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:03.326+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:03.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:03.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:03.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:03.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:03.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:03.442+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:03.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:03.442+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:03.841+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:04.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:03.842+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:04.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:04.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:04.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:04.284+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:04.284+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:04.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:04.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:04.467+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:04.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:04.467+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:04.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:04.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:04.842+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:05.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:04.843+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:05.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:05.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:05.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:05.313+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:05.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:05.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:05.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:05.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:05.464+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:05.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:05.464+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:06.104 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:05.980+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33552 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:06.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:05.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:06.104 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:05.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:06.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:05.820+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:06.105 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:05.820+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:06.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:06.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:06.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:06.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:06.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:06.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:06.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:06.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:06.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:06.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:06.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:06.492+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:06.816 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:06.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:06.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:06.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:07.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:06.814+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:07.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:06.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:07.507 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:07.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:07.507 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:07.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:07.508 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:07.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:07.508 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:07.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:07.814 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:07.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:07.814 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:07.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:07.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:07.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:07.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:07.507+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:07.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:07.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:07.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:08.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:07.815+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:08.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:08.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:08.321+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:08.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:08.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:08.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:08.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:08.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:08.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:08.515+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:09.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:09.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:08.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:08.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:08.808+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:09.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:08.809+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:09.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:09.247 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:18:09.247 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:18:09.248 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:18:09.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:09.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:09.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:09.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:09.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:09.554+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:09.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:09.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:09.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:09.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:09.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:09.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:09.857+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:10.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:09.859+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:10.526 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:10.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:10.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:10.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:10.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:10.358+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:10.527 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:10.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:10.814 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:10.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:10.814 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:10.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:10.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:10.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:10.815 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:10.526+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:11.108 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:10.984+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33642 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:11.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:10.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:11.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:10.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:11.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:10.813+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:11.109 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:10.814+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:11.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:11.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:11.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:11.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:11.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:11.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:11.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:11.557+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:11.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:11.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:11.851+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:12.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:11.852+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:12.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:12.516 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:12.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:12.517 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:12.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:12.517 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:12.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:12.517 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:12.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:12.839 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:12.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:12.839 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:12.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:12.839 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:12.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:12.839 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:12.516+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:12.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:12.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:12.838+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:13.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:12.839+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:13.394+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:13.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:13.394+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:13.549+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:13.549+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:13.549+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:13.550+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:13.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:13.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:13.850+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:14.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:13.851+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:14.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:14.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:14.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:14.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:14.537+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:14.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:14.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:14.347+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:14.347+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:14.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:14.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:15.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:14.885+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:15.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:14.885+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:15.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:15.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:15.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:15.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:15.570+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:15.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:15.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:15.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:15.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:15.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:15.989+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33732 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:15.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:16.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:15.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:15.923+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:16.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:15.923+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:16.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:16.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:16.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:16.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:16.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:16.545+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:16.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:16.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:16.411+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:16.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:16.412+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:16.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:16.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:16.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:17.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:16.920+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:17.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:17.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:17.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:17.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:17.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:17.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:17.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:17.497+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:17.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:17.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:17.452+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:17.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:17.453+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:18.015 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:17.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:18.015 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:17.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:18.015 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:17.928+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:18.015 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:17.929+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:18.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:18.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:18.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:18.419+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:18.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:18.420+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:18.527+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:18.527+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:18.527+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:18.528+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:19.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:18.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:19.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:18.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:19.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:18.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:19.422 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:18.964+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:19.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:19.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:19.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:19.421+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:19.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:19.422+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:19.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:19.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:19.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:19.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:19.500+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:20.388 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:19.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:20.388 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:19.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:20.389 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:19.998+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:20.389 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:19.999+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:20.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:20.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:20.465+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:20.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:20.466+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:20.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:20.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:20.387+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:20.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:20.389+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:21.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:21.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:21.416 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:20.993+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33823 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:21.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:20.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:21.416 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:20.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:21.417 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:20.958+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:21.417 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:20.959+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:21.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:21.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:21.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:21.415+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:21.416+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:21.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:21.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:21.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:21.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:21.493+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:21.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:21.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:22.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:21.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:22.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:21.912+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:22.416+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:22.416+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:22.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:22.416+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:22.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:22.416+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:22.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:22.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:22.491+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:22.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:22.492+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:23.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:22.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:22.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:22.890+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:23.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:22.890+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:23.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:23.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:23.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:23.378+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:23.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:23.379+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:23.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:23.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:23.483+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:23.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:23.483+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:23.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:24.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:23.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:23.846+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:24.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:23.847+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:24.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:24.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:24.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:24.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:24.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:24.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:24.387+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:24.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:24.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:24.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:24.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:24.460+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:24.821+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:25.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:24.822+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:25.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:25.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:25.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:25.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:25.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:25.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:25.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:25.361+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:25.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:25.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:25.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:25.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:25.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:25.462+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:25.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:25.462+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:26.119 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:25.997+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 33913 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:26.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:25.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:26.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:25.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:26.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:25.801+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:26.119 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:25.802+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:26.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:26.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:26.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:26.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:26.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:26.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:26.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:26.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:26.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:26.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:26.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:26.475+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:26.837 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:26.475+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:26.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:26.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:26.836+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:27.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:26.837+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:27.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:27.499+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:27.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:27.500+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:27.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:27.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:27.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:27.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:27.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:27.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:27.328+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:27.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:27.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:27.854+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:27.855+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:28.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:28.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:28.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:28.357+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:28.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:28.358+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:28.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:28.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:28.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:28.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:28.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:28.975 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:28.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.975 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:28.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.975 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:28.870+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:28.975 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:28.870+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:29.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:29.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:29.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:29.336+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:29.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:29.337+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:29.460+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:29.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:29.460+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:29.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:29.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:30.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:29.903+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:30.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:29.904+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:30.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:30.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:30.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:30.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:30.488 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:30.383+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:30.489 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:30.384+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:30.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:30.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:30.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:30.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:30.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:30.487+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:30.919 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:30.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:31.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:31.001+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34003 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:31.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:30.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:30.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:30.918+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:31.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:30.918+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:31.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:31.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:31.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:31.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:31.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:31.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:31.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:31.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:31.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:31.519+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:31.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:32.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:31.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:32.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:31.906+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:32.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:31.907+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:32.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:32.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:32.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:32.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:32.513+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:32.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:32.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:32.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:32.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:32.404+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:32.915+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:32.915+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:33.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:32.915+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:33.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:32.916+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:33.517 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:33.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:33.517 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:33.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:33.518 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:33.414+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:33.518 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:33.415+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:33.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:33.516+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:33.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:33.516+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:33.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:33.516+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:33.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:33.517+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:33.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:33.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:33.907+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:34.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:33.908+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:34.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:34.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:34.506+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:34.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:34.506+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:34.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:34.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:34.385+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:34.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:34.385+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:34.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:34.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:35.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:34.919+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:35.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:34.920+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:35.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:35.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:35.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:35.368+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:35.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:35.369+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:35.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:35.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:35.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:35.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:35.546+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:36.131 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:36.006+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34092 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:36.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:35.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:36.131 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:35.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:36.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:35.871+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:36.132 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:35.871+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:36.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:36.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:36.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:36.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:36.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:36.327+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:36.913 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:36.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:36.913 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:36.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:36.914 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:36.581+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:36.914 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:36.581+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:36.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:36.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:37.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:36.912+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:37.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:36.913+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:37.459 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:37.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:37.459 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:37.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:37.459 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:37.374+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:37.459 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:37.375+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:37.607+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:37.607+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:37.607+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:37.607+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:38.400 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:37.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:38.401 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:37.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:38.401 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:37.963+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:38.401 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:37.964+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:38.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:38.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:38.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:38.566+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:38.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:38.567+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:38.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:38.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:38.399+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:38.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:38.401+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:38.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:38.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:38.968+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:39.387 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:38.969+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:39.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:39.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:39.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:39.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:39.386+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:39.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:39.386+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:39.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:39.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:39.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:39.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:39.541+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:40.351 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:39.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:40.352 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:39.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:40.352 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:39.994+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:40.352 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:39.994+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:40.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:40.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:40.351+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:40.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:40.351+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:40.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:40.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:40.533+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:40.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:40.533+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:41.372 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:41.010+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34183 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:41.372 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:41.372 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:41.372 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.005+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:41.373 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.007+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:41.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:41.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:41.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:41.371+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:41.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:41.371+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:41.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:41.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:41.560+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:41.560+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:42.404 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:42.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:42.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.979+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:42.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:41.980+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:42.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:42.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:42.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:42.530 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:18:42.531 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:18:42.533 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053/remote/smithi026/crash 2024-03-24T11:18:42.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:18:42.564 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053/remote/smithi031/crash 2024-03-24T11:18:42.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:42.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:42.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:42.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:42.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:42.403+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:42.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:42.404+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:42.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:42.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:42.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:42.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:42.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:42.588+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:42.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:42.588+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:43.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:42.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:43.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:42.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:43.370 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:42.964+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:43.371 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:42.965+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:43.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:43.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:43.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:43.369+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:43.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:43.370+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:43.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:43.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:43.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:43.595+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:43.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:43.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:43.985+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:44.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:43.986+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:44.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:44.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:44.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:44.377+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:44.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:44.378+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:44.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:44.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:44.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:44.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:44.704 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:44.587+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:45.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:44.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:45.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:44.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:45.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:44.983+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:45.425 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:44.984+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:45.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:45.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:45.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:45.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:45.423+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:45.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:45.425+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:45.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:45.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:45.567+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:45.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:45.567+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:46.377 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:46.014+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34271 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:46.377 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:45.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:46.377 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:45.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:46.377 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:45.990+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:46.377 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:45.991+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:46.535+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:46.535+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:46.535+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:46.535+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:46.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:46.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:46.376+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:46.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:46.377+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:47.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:47.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:47.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:47.025+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:47.405 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:47.026+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:47.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:47.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:47.573+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:47.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:47.573+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:47.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:47.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:47.404+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:47.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:47.405+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:48.356 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:48.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:48.357 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:48.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:48.357 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:48.066+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:48.357 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:48.067+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:48.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:48.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:48.526+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:48.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:48.527+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:48.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:48.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:48.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:48.356+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:48.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:48.356+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:49.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:49.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:49.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:49.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:49.330 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:49.070+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:49.331 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:49.070+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:49.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:49.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:49.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:49.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:49.327+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:49.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:49.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:49.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:49.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:49.572+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:50.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:50.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:50.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:50.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:50.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:50.091+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:50.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:50.091+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:50.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:50.091+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:50.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:50.092+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:50.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:50.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:50.540+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:50.540+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:51.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:51.019+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34353 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:51.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:51.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:51.125+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:51.125+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:51.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:51.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:51.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:51.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:51.265+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:51.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:51.266+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:51.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:51.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:51.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:51.494+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:51.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:51.494+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:52.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:52.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:52.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:52.129+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:52.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:52.130+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:52.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:52.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:52.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:52.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:52.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:52.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:52.496+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:52.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:52.496+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:53.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:53.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:53.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:53.102+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:53.103+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:53.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:53.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:53.286+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:53.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:53.287+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:53.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:53.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:53.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:53.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:53.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:53.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:54.442 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:54.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:54.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:54.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:54.151+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:54.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:54.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:54.443 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:54.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:54.444 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:54.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:54.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:18:54.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:18:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:54.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:54.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:54.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:54.441+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:54.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:54.442+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:55.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:55.192+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:55.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:55.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:55.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:55.305+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:55.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:55.305+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:55.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:55.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:55.488+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:55.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:55.488+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:56.293 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:18:56.023+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34441 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:18:56.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:56.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:56.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:56.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:56.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:56.176+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:56.294 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:56.177+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:56.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:56.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:56.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:56.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:56.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:56.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:56.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:56.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:56.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:56.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:56.537+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:57.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:57.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:57.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:57.299+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:57.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:57.300+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:57.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:57.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:57.150+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:57.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:57.151+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:57.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:18:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:57.541+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:57.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:57.541+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:57.541+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:57.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:57.541+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:58.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:58.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:58.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:58.278+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:58.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:58.278+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:58.136+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:58.136+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:58.136+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:58.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:58.136+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:58.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:58.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:58.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:58.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:58.516+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:59.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:59.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:59.098+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:18:59.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:18:59.099+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:59.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:59.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:59.266+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:18:59.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:18:59.267+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:18:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:59.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:59.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:59.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:59.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:18:59.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:18:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:18:59.548+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:00.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:00.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:00.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:00.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:00.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:00.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:00.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:00.101+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:00.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:00.102+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:00.910 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:00.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:00.910 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:00.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:00.910 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:00.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:00.911 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:00.597+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:01.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:01.027+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34533 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:01.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:01.201 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:01.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:01.137+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:01.202 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:01.138+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:01.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:01.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:01.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:01.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:01.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:01.262+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:01.583 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:01.262+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:01.582+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:01.582+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:01.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:01.582+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:01.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:01.582+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:02.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:02.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:02.140+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:02.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:02.141+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:02.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:02.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:02.308+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:02.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:02.309+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:02.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:02.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:02.599+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:02.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:02.599+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:03.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:03.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:03.163+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:03.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:03.164+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:03.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:03.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:03.296+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:03.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:03.297+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:03.850 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:03.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:03.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:03.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:03.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:03.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:03.851 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:03.571+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:04.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:04.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:04.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:04.212+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:04.584 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:04.212+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:04.585 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:04.585 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:04.585 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:04.585 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:04.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:04.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:04.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:04.583+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:04.584+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:05.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:05.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:05.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:05.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:05.610 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:05.252+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:05.611 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:05.252+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:05.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:05.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:05.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:05.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:05.612 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:05.314+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:05.612 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:05.314+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:05.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:05.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:05.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:05.609+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:06.300 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:06.032+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34622 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:06.300 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:06.281+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:06.300 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:06.281+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:06.301 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:06.281+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:06.301 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:06.281+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:06.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:06.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:06.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:06.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:06.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:06.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:06.611 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:06.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:06.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:06.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:06.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:06.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:06.954 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:06.609+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:06.954 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:06.610+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:07.645 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:07.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:07.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:07.326+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:07.326+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:07.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:07.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:07.646 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:07.268+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:07.647 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:07.269+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:07.643+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:07.643+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:07.643+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:07.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:07.644+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:08.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:08.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:08.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:08.364+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:08.365+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:08.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:08.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:08.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:08.242+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:08.630 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:08.243+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:08.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:08.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:08.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:08.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:08.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:09.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:09.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:09.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:09.195+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:09.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:09.196+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:09.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:09.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:09.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:09.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:09.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:09.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:09.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:09.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:09.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:09.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:09.610+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:09.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:09.610+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:10.653 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:10.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:10.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:10.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:10.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:10.241+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:10.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:10.241+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:10.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:10.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:10.654 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:10.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:10.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:10.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:10.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:10.292+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:10.653+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:10.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:10.653+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:11.337 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:11.036+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34713 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:11.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:11.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:11.337 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:11.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:11.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:11.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:11.338 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:11.193+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:11.618 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:11.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:11.619 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:11.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:11.619 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:11.335+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:11.619 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:11.336+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:11.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:11.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:11.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:11.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:11.616+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:11.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:11.617+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:12.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:12.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:12.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:12.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:12.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:12.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:12.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:12.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:12.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:12.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:12.157+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:12.453 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:12.157+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:12.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:12.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:12.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:12.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:12.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:13.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:13.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:13.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:13.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:13.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:13.167+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:13.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:13.168+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:13.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:13.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:13.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:13.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:13.353+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:13.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:13.353+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:13.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:13.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:13.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:13.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:13.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:13.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:13.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:13.605+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:14.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:14.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:14.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:14.193+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:14.194+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:14.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:14.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:14.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:14.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:14.320+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:14.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:14.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:14.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:14.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:14.626+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:15.656 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:15.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:15.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:15.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:15.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:15.232+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:15.657 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:15.233+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:15.657 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:15.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:15.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:15.280+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:15.658 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:15.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:15.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:15.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:15.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:15.838 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:19:15.839 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:19:15.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-24T11:19:15.874 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:19:15.874 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:19:15.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:15.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:15.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:15.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:15.656+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:15.939 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:15.656+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:15.941 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:19:15.941 INFO:teuthology.orchestra.run.smithi026.stderr:: No such file or directory 2024-03-24T11:19:15.942 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mon.smithi026.log 2024-03-24T11:19:15.943 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.dfxoad.log 2024-03-24T11:19:15.943 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose --/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mon.smithi026.log: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:19:15.943 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.dfxoad.log: gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:19:15.943 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.ryhyqm.log 2024-03-24T11:19:15.944 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mgr.smithi026.bbjcru.log 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.ryhyqm.log: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mgr.smithi026.bbjcru.log: gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr: 90.6% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:19:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:19:15.948 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.log: 86.9% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:19:15.948 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:19:15.954 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:19:15.954 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi026.log 2024-03-24T11:19:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:19:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi026.log: 94.6% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi026.log.gz 2024-03-24T11:19:15.980 INFO:teuthology.orchestra.run.smithi026.stderr: 95.0% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.dfxoad.log.gz 2024-03-24T11:19:16.201 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:16.041+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34802 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:16.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:16.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:16.659 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:16.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:16.228+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:16.229+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:16.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:16.660 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:16.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:16.833 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mgr.smithi026.bbjcru.log: file size changed while zipping 2024-03-24T11:19:16.834 INFO:teuthology.orchestra.run.smithi026.stderr: 89.9% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mgr.smithi026.bbjcru.log.gz 2024-03-24T11:19:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:16.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:16.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:16.658+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:16.658+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mon.smithi026.log: file size changed while zipping 2024-03-24T11:19:17.434 INFO:teuthology.orchestra.run.smithi026.stderr: 92.6% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mon.smithi026.log.gz 2024-03-24T11:19:17.436 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-24T11:19:17.436 INFO:teuthology.orchestra.run.smithi026.stderr: 91.1% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:19:17.653 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:17.235+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:17.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:17.235+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:17.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:17.235+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:17.654 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:17.236+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:17.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:17.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:17.652+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:17.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:17.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:17.655 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:17.329+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:17.656 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:17.331+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:17.653+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:18.535 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-24T11:19:18.535 INFO:teuthology.orchestra.run.smithi026.stderr: 90.8% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:18.273+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:18.273+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:18.273+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:18.274+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:18.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:18.675 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:18.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:18.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:18.363+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:18.676 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:18.364+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:18.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:18.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:18.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:18.674+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:18.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:18.674+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:18.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:19.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:19.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:19.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:19.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:19.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:19.304+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:19.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:19.304+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:19.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:19.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:19.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:19.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:19.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:19.372+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:19.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:19.373+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:19.629 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:19.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:19.630 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:19.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:19.630 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:19.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:19.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:20.564 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-24T11:19:20.565 INFO:teuthology.orchestra.run.smithi026.stderr: 95.2% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:19:20.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:20.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:20.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:20.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:20.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:20.296+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:20.594 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:20.297+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:20.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:20.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:20.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:20.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:20.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:20.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:20.595 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:20.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:20.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:20.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:20.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:20.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:20.593+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:21.317 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:21.046+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34894 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:21.631 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:21.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:21.631 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:21.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:21.631 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:21.316+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:21.631 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:21.317+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:21.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:21.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:21.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:21.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:21.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:21.360+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:21.632 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:21.360+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:21.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:21.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:21.630+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:21.631+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:22.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:22.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:22.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:22.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:22.628 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:22.337+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:22.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:22.338+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:22.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:22.357+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:22.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:22.357+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:22.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:22.357+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:22.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:22.357+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:22.630 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:22.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:22.630 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:22.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:22.630 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:22.627+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:22.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:22 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:22.627+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:23.628 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:23.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:23.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:23.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:23.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:23.404+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:23.629 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:23.405+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:23.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:23.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:23.629 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:23.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:23.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:23.310+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:23.630 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:23.311+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:23.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:23.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:23.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:23.628+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:23.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:23 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:23.628+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:24.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:24.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:24.679 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:24.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:24.431+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:24.432+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:24.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:24.680 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:24.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:24.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:24.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:24.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:24.678+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:24.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:24 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:24.678+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:25.652 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:25.409+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:25.652 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:25.409+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:25.652 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:25.409+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:25.652 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:25.411+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:25.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:25.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:25.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:25.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:25.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:25.330+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:25.653 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:25.331+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:25.651+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:25.651+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:25.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:25.651+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:25.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:25 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:25.651+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:25.993 INFO:teuthology.orchestra.run.smithi026.stderr: 92.4% -- replaced with /var/log/ceph/e07c400a-e9c9-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi026.ryhyqm.log.gz 2024-03-24T11:19:25.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-24T11:19:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m10.066s 2024-03-24T11:19:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m20.635s 2024-03-24T11:19:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.502s 2024-03-24T11:19:26.371 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:26.050+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 34983 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:26.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:26.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:26.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:26.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:26.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:26.449+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:26.643 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:26.449+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:26.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:26.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:26.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:26.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:26.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:26.370+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:26.644 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:26.371+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:26.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:26.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:26.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:26.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:26.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:26 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:26.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:27.596 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:27.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:27.596 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:27.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:27.596 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:27.493+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:27.596 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:27.494+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:27.596 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:27.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:27.597 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:27.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:27.597 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:27.343+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:27.597 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:27.343+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:27.873 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:27.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:27.873 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:27.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:27.873 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:27.595+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:27.873 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:27 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:27.595+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:27.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:27.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:28.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:28.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:28.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:28.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:28.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:28.467+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:28.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:28.468+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:28.606 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:28.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:28.606 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:28.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:28.606 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:28.317+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:28.606 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:28.317+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:28.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:28.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:28.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:28.604+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:28.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:28 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:28.605+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:29.558 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:29.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:29.558 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:29.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:29.558 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:29.504+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:29.559 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:29.505+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:29.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:29.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:29.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:29.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:29.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:29.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:29.559 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:29.289+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:29.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:29.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:29.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:29.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:29 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:29.558+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:30.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:30.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:30.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:30.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:30.519 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:30.289+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:30.520 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:30.290+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:30.547+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:30.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:30.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:30.548+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:30.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:30.548+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:30.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:30.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:30.518+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:30.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:30 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:30.519+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:31.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:31.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:31.055+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 35070 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:31.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:31.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:31.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:31.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:31.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:31.291+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:31.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:31.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:31.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:31.486+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:31.487+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:31.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:31.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:31.593+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:31.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:31 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:31.593+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:32.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:32.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:32.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:32.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:32.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:32.494+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:32.589 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:32.495+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:32.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:32.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:32.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:32.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:32.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:32.331+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:32.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:32.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:32.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:32.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:32.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:32.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:32.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:32.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:32.912 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:32 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:32.588+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:33.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:33.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:33.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:33.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:33.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:33.461+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:33.605 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:33.462+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:33.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:33.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:33.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:33.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:33.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:33.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:33.605 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:33.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:33.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:33.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:33.603+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:33.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:33 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:33.604+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:34.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:34.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:34.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:34.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:34.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:34.480+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:34.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:34.481+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:34.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:34.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:34.328+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:34.329+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:34.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:34.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:34.587+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:34.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:34 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:34.588+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:35.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:35.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:35.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:35.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:35.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:35.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:35.626 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:35.480+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:35.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:35.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:35.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:35.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:35.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:35.298+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:35.627 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:35.299+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:35.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:35.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:35.625+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:35.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:35 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:35.625+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:36.451 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:36.060+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 35159 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:36.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:36.451 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:36.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:36.294+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:36.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:36.295+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:36.951 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:36.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:36.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:36.513+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:36.952 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:36.514+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:36.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:36.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:36.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:36.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:36.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:36.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:36.953 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:36 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:36.589+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:37.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:37.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:37.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:37.521+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:37.521+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:37.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:37.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:37.548 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:37.293+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:37.549 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:37.294+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:37.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:37.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:37.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:37.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:37.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:37 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:37.547+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:38.569 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:38.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:38.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:38.260+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:38.261+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:38.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:38.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:38.570 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:38.478+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:38.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:38.478+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:38.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:38.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:38.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:38.568+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:38.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:38 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:38.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:39.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:39.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:39.545 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:39.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:39.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:39.271+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:39.546 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:39.272+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:39.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:39.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:39.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:39.433+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:39.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:39.434+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:39.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:39.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:39.544+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:39.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:39 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:39.545+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:40.573 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:40.407+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:40.407+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:40.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:40.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:40.574 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:40.252+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:40.575 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:40.253+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:40.846 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:40.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:40.846 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:40.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:40.846 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:40.572+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:40.846 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:40 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:40.573+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:41.188 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:41.064+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 35251 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:41.441 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:41.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:41.442 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:41.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:41.442 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:41.263+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:41.442 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:41.264+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:41.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:41.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:41.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:41.440+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:41.441+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:41.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:41.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:41.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:41.557+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:41.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:41 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:41.557+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:42.576 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:42.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:42.576 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:42.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:42.576 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:42.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:42.576 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:42.240+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:42.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:42.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:42.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:42.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:42.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:42.464+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:42.577 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:42.464+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:42.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:42.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:42.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:42.575+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:42.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:42 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:42.575+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:43.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:43.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:43.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:43.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:43.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:43.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:43.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:43.203+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:43.425 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:43.203+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:43.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:43.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:43.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:43.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:43.424+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:43.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:43.424+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:43.528+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:43.528+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:43.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:43.528+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:43.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:43 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:43.528+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:44.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:44.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:44.398+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:44.571 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:44.399+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:44.571 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:44.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:44.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:44.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:44.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:44.239+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:44.572 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:44.240+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:44.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:44.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:44.570+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:44.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:44 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:44.570+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:45.598 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:45.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:45.598 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:45.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:45.393+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:45.393+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:45.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:45.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:45.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:45.599 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:45.276+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:45.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:45.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:45.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:45.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:45.597+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:45.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:45 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:45.598+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:46.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:46.395 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:46.069+0000 7f976402b700 -1 mon.smithi026@0(probing) e2 get_health_metrics reporting 35340 slow ops, oldest is monmgrreport(gid 14215, 0 checks, 1 progress events) 2024-03-24T11:19:46.395 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:46.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:46.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:46.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:46.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:46.276+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:46.396 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:46.277+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:46.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:46.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:46.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:46.642+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:46.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:46.642+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:46.394+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:46.394+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:46.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:46.394+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:46.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:46 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:46.395+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:47.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:47.388+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:47.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:47.388+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:47.615 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:47.388+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:47.616 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:47.389+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:47.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:47.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:47.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:47.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:47.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:47.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:47.616 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:47.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:47.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:47.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:47.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:47.614+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:47.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:47 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:47.615+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:48.636 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:48.385+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:48.385+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:48.385+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:48.386+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:48.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:48.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:48.637 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:48.295+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:48.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:48.296+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:48.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:48.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:48.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:48.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:48 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:48.636+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:49.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:19:49.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:49.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:19:49.182 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:19:49.183 INFO:tasks.cephadm:Removing cluster... 2024-03-24T11:19:49.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 --force 2024-03-24T11:19:49.332 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: e07c400a-e9c9-11ee-95cd-87774f69a715 2024-03-24T11:19:49.637 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:49 smithi026 systemd[1]: Stopping Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T11:19:49.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:49.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:49.637 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:49.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:49.402+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:49.402+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:49.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:49.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:49.291+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:49.638 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:49.291+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:49.954 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:49.693+0000 7f9767031700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi026 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:19:49.954 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026[29206]: 2024-03-24T11:19:49.693+0000 7f9767031700 -1 mon.smithi026@0(probing) e2 *** Got Signal Terminated *** 2024-03-24T11:19:49.955 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:49.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:49.955 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:49.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:49.955 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:49.636+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:49.955 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:49 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:49.636+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:50.296 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:49 smithi026 podman[85855]: 2024-03-24 11:19:49.989161756 +0000 UTC m=+0.428674085 container died dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T11:19:50.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 podman[85855]: 2024-03-24 11:19:50.395032089 +0000 UTC m=+0.834544416 container cleanup dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T11:19:50.603 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 bash[85855]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026 2024-03-24T11:19:50.603 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:50.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:50.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:50.417+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:50.418+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:50.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:50.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:50.324+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:50.604 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:50.325+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:50.951 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 podman[85869]: 2024-03-24 11:19:50.754960743 +0000 UTC m=+0.769118946 container remove dfd13627777b5f7960fd41d798be15e492975a0045f48737fd6321fa0a91e4e5 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-mon-smithi026, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:19:50.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:50.602+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:50.602+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:50.602+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:50.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:50 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:50.602+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:51.323 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service: Deactivated successfully. 2024-03-24T11:19:51.324 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 systemd[1]: Stopped Ceph mon.smithi026 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T11:19:51.324 INFO:journalctl@ceph.mon.smithi026.smithi026.stdout:Mar 24 11:19:50 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@mon.smithi026.service: Consumed 25.941s CPU time. 2024-03-24T11:19:51.607 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:51.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:51.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:51.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:51.453+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:51.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:51.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:51.608 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:51.322+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:51.609 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:51.322+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:51.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:51.606+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:51.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:51.606+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:51.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:51.606+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:51.892 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:51 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:51.607+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:52.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:52.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:52.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:52.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:52.458+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:52.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:52.458+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:52.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:52.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:52.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:52.354+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:52.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:52.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:52.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:52.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:52.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:52.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:52.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:52.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:52.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:52 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:52.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:53.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:53.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:53.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:53.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:53.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:53.402+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:53.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:53.403+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:53.664 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:53.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:53.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:53.601+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:53.601+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:53.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:53.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:53.444+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:53.665 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:53 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:53.444+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:54.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:54.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:54.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:54.452+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:54.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:54.453+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:54.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:54.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:54.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:54.382+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:54.383+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:54.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:54.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:54.600+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:54.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:54 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:54.601+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:55.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:55.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:55.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:55.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:55.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:55.338+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:55.591 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:55.339+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:55.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:55.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:55.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:55.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:55.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:55.421+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:55.592 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:55.422+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:55.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:55.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:55.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:55.589+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:55.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:55 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:55.590+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:56.588 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:56.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:56.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:56.435+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:56.436+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:56.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:56.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:56.589 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:56.355+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:56.590 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:56.355+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:56.590 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:56.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:56.590 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:56.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:56.590 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:56.561+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:56.590 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:56 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:56.561+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:57.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:57.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:57.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:57.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:57.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:57.454+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:57.520 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:57.455+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:57.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:57.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:57.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:57.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:57.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:57.312+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:57.521 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:57.313+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:57.770 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:57.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:57.771 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:57.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:57.771 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:57.519+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:57.771 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:57 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:57.520+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:58.663 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:58.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:58.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:58.320+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:58.321+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:58.432+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:58.432+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:58.432+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:58.664 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:58.433+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:58.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:58.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:58.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:58.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:58.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:58.569+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:58.665 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:58 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:58.569+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:59.533 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:59.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:59.534 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:59.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:59.534 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:59.412+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:19:59.534 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:19:59.412+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:59.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:59.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:59.534 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:59.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:59.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:59.281+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:19:59.535 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:19:59.282+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:19:59.936 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:59.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:59.936 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:59.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:59.936 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:59.532+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:19:59.936 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:19:59 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:19:59.533+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:00.564 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:00.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:00.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:00.251+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:00.252+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 systemd[1]: Stopping Ceph osd.0 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.364+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:00.565 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.364+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.364+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.365+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.563+0000 7fce2d88c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.563+0000 7fce2d88c700 -1 osd.0 47 *** Got signal Terminated *** 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:00.563+0000 7fce2d88c700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:20:00.566 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:00.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:00.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:00.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:00.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:00.493+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:00.567 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:00 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:00.493+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:01.376 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:01.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:01.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:01.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:01.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:01.297+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:01.377 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:01.298+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:01.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:01.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:01.375+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:01.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:01.376+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:01.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:01.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:01.518+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:01.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:01 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:01.518+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:02.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:02.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:02.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:02.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:02.546 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:02.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:02.547 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:02.345+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:02.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:02.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:02.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:02.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:02.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:02.249+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:02.547 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:02.249+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:02.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:02.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:02.545+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:02.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:02 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:02.545+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:03.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:03.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:03.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:03.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:03.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:03.538+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:03.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:03.538+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:03.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:03.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:03.292+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:03.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:03.293+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:03.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:03.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:03.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:03.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:03.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:03.374+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:03.703 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:03 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:03.374+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:04.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:04.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:04.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:04.290+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:04.543 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:04.291+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:04.544 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:04.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:04.544 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:04.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:04.544 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:04.389+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:04.544 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:04.389+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:04.542+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:04.542+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:04.542+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:04.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:04 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:04.542+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:05.586 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:05.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:05.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:05.344+0000 7fce2867f700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:33.075803+0000 (oldest deadline 2024-03-24T11:14:53.075803+0000) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0[48036]: 2024-03-24T11:20:05.345+0000 7fce2867f700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:05.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:05.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:05.587 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:05.248+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:05.588 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:05.249+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:05.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:05.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:05.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:05.585+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:05.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:05 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:05.586+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:06.364 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:06 smithi026 podman[86823]: 2024-03-24 11:20:06.112313922 +0000 UTC m=+5.664633906 container died 6dd3171075512bd6538cce3d0159cb0de487c940410fc20ef39b28c467ce4c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=reef-738c2fd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:20:06.365 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:06.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:06.365 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:06.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:06.365 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:06.257+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:06.365 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:06.257+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:06.701 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:06 smithi026 podman[86823]: 2024-03-24 11:20:06.504502357 +0000 UTC m=+6.056822338 container cleanup 6dd3171075512bd6538cce3d0159cb0de487c940410fc20ef39b28c467ce4c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T11:20:06.702 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:06 smithi026 bash[86823]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:20:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:06.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:06.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:06.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:06.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:06 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:06.547+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:07.048 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:06 smithi026 podman[86845]: 2024-03-24 11:20:06.757183916 +0000 UTC m=+0.640594549 container remove 6dd3171075512bd6538cce3d0159cb0de487c940410fc20ef39b28c467ce4c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-738c2fd, org.label-schema.build-date=20240319) 2024-03-24T11:20:07.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:07.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:07.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:07.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:07.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:07.268+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:07.316 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:07.269+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:07.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:06.951557348 +0000 UTC m=+0.017870402 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T11:20:07.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:07.058343372 +0000 UTC m=+0.124656443 container create cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:20:07.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:07.24432645 +0000 UTC m=+0.310639497 container init cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T11:20:07.317 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:07.247524185 +0000 UTC m=+0.313837222 container start cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:20:07.568 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:07.319177311 +0000 UTC m=+0.385490355 container attach cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=reef-738c2fd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T11:20:07.568 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:07.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:07.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:07.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:07.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:07.525+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:07.569 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:07 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:07.525+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:07.855 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:07 smithi026 podman[86935]: 2024-03-24 11:20:07.56877583 +0000 UTC m=+0.635088848 container died cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:20:08.451 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:08 smithi026 podman[86970]: 2024-03-24 11:20:08.114539831 +0000 UTC m=+0.540656229 container remove cbef51e6ec7a4dd73ce8cf7a289b6b1c2c1efa8c2068ffb9a87de30ca5b40b6a (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=reef-738c2fd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:08 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:08 smithi026 systemd[1]: Stopped Ceph osd.0 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 24 11:20:08 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.0.service: Consumed 17min 51.088s CPU time. 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:08.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.306+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.307+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:08.453 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 systemd[1]: Stopping Ceph osd.1 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T11:20:08.951 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.592+0000 7f19baa32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.592+0000 7f19baa32700 -1 osd.1 47 *** Got signal Terminated *** 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:08.592+0000 7f19baa32700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:08.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:08.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:08.484+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:08.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:08 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:08.485+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:09.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:09.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:09.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:09.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:09.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:09.522+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:09.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:09.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:09.339+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:09.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:09 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:09.340+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:10.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:10.332+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:10.332+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:10.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:10.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:10.511+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:10.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:10 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:10.511+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:11.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:11.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:11.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:11.304+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:11.304+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:11.515+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:11.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:11 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:11.515+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:12.553 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:12.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:12.554 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:12.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:12.554 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:12.302+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:12.554 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:12.303+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:12.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:12.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:12.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:12.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:12.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:12 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:12.553+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:13.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:13.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:13.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:13.546+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:13.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:13.546+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:13.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:13.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:13.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:13.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:13.327+0000 7f19b7829700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:39.044292+0000 (oldest deadline 2024-03-24T11:14:59.044292+0000) 2024-03-24T11:20:13.703 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:13 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1[55511]: 2024-03-24T11:20:13.328+0000 7f19b7829700 -1 osd.1 47 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.14536.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:14.392 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:14 smithi026 podman[87051]: 2024-03-24 11:20:14.085731255 +0000 UTC m=+5.619031565 container died 8bf2a3e7910dd53fd6a6a7a983026f93aa08334306eb7a1274ae86f8f74bbc29 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:20:14.393 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:14 smithi026 podman[87051]: 2024-03-24 11:20:14.392959637 +0000 UTC m=+5.926259944 container cleanup 8bf2a3e7910dd53fd6a6a7a983026f93aa08334306eb7a1274ae86f8f74bbc29 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:20:14.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:14 smithi026 bash[87051]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:20:14.702 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:14 smithi026 podman[87068]: 2024-03-24 11:20:14.619229612 +0000 UTC m=+0.534263049 container remove 8bf2a3e7910dd53fd6a6a7a983026f93aa08334306eb7a1274ae86f8f74bbc29 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T11:20:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:14.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:14.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:14.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:14.522+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:14.703 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:14 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:14.522+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:15.339 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:14.965482419 +0000 UTC m=+0.018651256 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T11:20:15.339 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:15.070129464 +0000 UTC m=+0.123298335 container create cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:20:15.340 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:15.255756103 +0000 UTC m=+0.308924943 container init cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T11:20:15.340 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:15.258713257 +0000 UTC m=+0.311882085 container start cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T11:20:15.340 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:15.339017734 +0000 UTC m=+0.392186571 container attach cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T11:20:15.701 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:15 smithi026 podman[87160]: 2024-03-24 11:20:15.555968381 +0000 UTC m=+0.609137224 container died cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T11:20:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:15.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:15.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:15.565+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:15.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:15 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:15.565+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:16.147 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:16 smithi026 podman[87186]: 2024-03-24 11:20:16.126095892 +0000 UTC m=+0.565274225 container remove cc88f6dabb582c675da105a423990581ca085ab80fc03893b22612b715bdd0a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:20:16.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 systemd[1]: Stopping Ceph osd.2 for e07c400a-e9c9-11ee-95cd-87774f69a715... 2024-03-24T11:20:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:16 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T11:20:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:16 smithi026 systemd[1]: Stopped Ceph osd.1 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T11:20:16.452 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 24 11:20:16 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.1.service: Consumed 20.246s CPU time. 2024-03-24T11:20:16.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.536+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.536+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.586+0000 7fee234e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.586+0000 7fee234e6700 -1 osd.2 47 *** Got signal Terminated *** 2024-03-24T11:20:16.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:16 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:16.586+0000 7fee234e6700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:20:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:17.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:17.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:17.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:17.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:17.553+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:17.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:17 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:17.553+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:18.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:18.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:18.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:18.578+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:18.702 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:18 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:18.578+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:19.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:19.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:19.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:19.534+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:19.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:19 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:19.534+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:20.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:20.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:20.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:20.502+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:20.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:20 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:20.502+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:21.951 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:21.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:21.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:21.512+0000 7fee1f2db700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.31:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:14:28.876173+0000 (oldest deadline 2024-03-24T11:14:48.876173+0000) 2024-03-24T11:20:21.952 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:21 smithi026 ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2[63384]: 2024-03-24T11:20:21.512+0000 7fee1f2db700 -1 osd.2 47 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.14536.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-24T11:20:22.338 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:22 smithi026 podman[87264]: 2024-03-24 11:20:22.079416134 +0000 UTC m=+5.585656011 container died d9c7f3ffbeae52284ab41d13451a128eb6f43190742ffd686aaff8c241fa1ccb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T11:20:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:22 smithi026 podman[87264]: 2024-03-24 11:20:22.437358359 +0000 UTC m=+5.943598236 container cleanup d9c7f3ffbeae52284ab41d13451a128eb6f43190742ffd686aaff8c241fa1ccb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T11:20:22.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:22 smithi026 bash[87264]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:20:23.055 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:22 smithi026 podman[87282]: 2024-03-24 11:20:22.705039943 +0000 UTC m=+0.624973416 container remove d9c7f3ffbeae52284ab41d13451a128eb6f43190742ffd686aaff8c241fa1ccb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T11:20:23.341 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:22.95853382 +0000 UTC m=+0.018208729 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T11:20:23.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:23.081086331 +0000 UTC m=+0.140761234 container create 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:20:23.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:23.27508743 +0000 UTC m=+0.334762343 container init 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T11:20:23.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:23.27823226 +0000 UTC m=+0.337907173 container start 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-738c2fd, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:20:23.342 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:23.341745836 +0000 UTC m=+0.401420743 container attach 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T11:20:23.701 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:23 smithi026 podman[87369]: 2024-03-24 11:20:23.573385371 +0000 UTC m=+0.633060275 container died 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:20:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:24 smithi026 podman[87396]: 2024-03-24 11:20:24.120164453 +0000 UTC m=+0.538657090 container remove 1877fddd0b5d26750f95a27f860b6709a3a419c6d04dd3358a2226cafb75e9a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-e07c400a-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:20:24.451 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:24 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T11:20:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:24 smithi026 systemd[1]: Stopped Ceph osd.2 for e07c400a-e9c9-11ee-95cd-87774f69a715. 2024-03-24T11:20:24.452 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 24 11:20:24 smithi026 systemd[1]: ceph-e07c400a-e9c9-11ee-95cd-87774f69a715@osd.2.service: Consumed 18.146s CPU time. 2024-03-24T11:20:30.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:20:30.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:20:34.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:20:43.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:20:43.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:20:46.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:20:58.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:20:58.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:01.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:04.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:04.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:04.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:04.222 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:21:04.223 INFO:tasks.cephadm:Teardown complete 2024-03-24T11:21:04.223 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:21:04.244 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T11:21:04.259 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:21:04.274 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T11:21:04.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T11:21:04.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:04.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:07.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:16.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:16.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:19.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:31.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:31.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:34.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:37.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:37.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:37.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:37.570 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:21:37.620 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:21:37.638 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:21:37.649 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:21:37.650 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:21:37.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:37.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:37.672 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:21:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:21:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-24T11:21:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 93 +673us[ +635us] +/- 81ms 2024-03-24T11:21:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 8 377 26 -89us[ -130us] +/- 76ms 2024-03-24T11:21:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -531us[ -573us] +/- 84ms 2024-03-24T11:21:37.678 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:21:40.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:21:49.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:21:49.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:52.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:04.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:04.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:07.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:10.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:10.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:10.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:10.942 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:22:10.943 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:22:10.959 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:22:10.971 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:22:10.971 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:22:11.041 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:22:11.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:11.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:14.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:23.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:23.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:26.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:38.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:38.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:41.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:44.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:44.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:44.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:44.418 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:22:44.419 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:22:44.428 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:22:44.441 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:22:44.500 INFO:teuthology.task.internal:Duration was 3403.749850 seconds 2024-03-24T11:22:44.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:22:44.592 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:22:44.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:22:44.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:44.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:22:47.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:22:56.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:22:56.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:59.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:11.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:11.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:14.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:17.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:17.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:17.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:17.891 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:23:17.891 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:23:17.907 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:23:17.919 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:23:17.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:23:17.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:17.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:21.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:30.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:30.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:33.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:45.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:45.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:48.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:51.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:51.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:51.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:23:51.234 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:23:51.235 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:23:51.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:23:51.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:51.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:23:51.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:51.302 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-03-24T11:23:54.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:03.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:03.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:06.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:18.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:18.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:21.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:24.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:24.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:24.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:24.543 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:24:24.543 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:24:24.560 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T11:24:24.574 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T11:24:24.574 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053/remote/smithi026 2024-03-24T11:24:24.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:24:24.617 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053/remote/smithi031 2024-03-24T11:24:24.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:24.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:27.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:36.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:36.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:39.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:51.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:51.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:54.818 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:57.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:57.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:57.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:24:57.887 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:24:57.887 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:24:57.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T11:24:57.922 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T11:24:57.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T11:24:58.001 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T11:24:58.001 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:24:58.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:24:58.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:58.024 INFO:teuthology.orchestra.run.smithi026.stdout: 265305 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 11:21 /home/ubuntu/cephtest 2024-03-24T11:24:58.024 INFO:teuthology.orchestra.run.smithi026.stdout: 265306 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 24 11:23 /home/ubuntu/cephtest/archive 2024-03-24T11:24:58.024 INFO:teuthology.orchestra.run.smithi026.stdout: 265310 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:25 /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:24:58.024 INFO:teuthology.orchestra.run.smithi026.stdout: 265312 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 10:25 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:24:58.024 INFO:teuthology.orchestra.run.smithi026.stdout: 265311 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 10:25 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:24:58.025 INFO:teuthology.orchestra.run.smithi026.stdout: 400373 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:22 /home/ubuntu/cephtest/archive/audit 2024-03-24T11:24:58.025 INFO:teuthology.orchestra.run.smithi026.stdout: 426269 180 -rw------- 1 ubuntu ubuntu 180495 Mar 24 11:22 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-24T11:24:58.025 INFO:teuthology.orchestra.run.smithi026.stdout: 265396 212 -rwxr-xr-x 1 ubuntu ubuntu 215105 Mar 24 10:32 /home/ubuntu/cephtest/cephadm 2024-03-24T11:24:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T11:25:01.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:25:10.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:25:10.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:13.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:25:25.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:25:25.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:28.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:25:31.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-24T11:25:31.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:31.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-24T11:25:31.298 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:25:31.299 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi031 2024-03-24T11:25:31.315 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T11:25:31.328 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T11:25:31.416 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-24T11:25:31.417 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T11:25:31.449 DEBUG:teuthology.nuke:shortname: smithi026 2024-03-24T11:25:31.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:25:31.477 DEBUG:teuthology.nuke:shortname: smithi031 2024-03-24T11:25:31.477 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:25:31.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:19.204852', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXzipRrH1SrY8FYg9ynf3fYPhWZ9oKyl/rry103egfreIS99qJD5Kob2MXPsPUJxBAw7U47PlFSUEY97xPsn/A='} 2024-03-24T11:25:31.504 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:19.206375', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIArAC655sYkrF00LrnFlueHTSqYgeStBPd1xrTiOorhQQoDRABWhPCf8swRSnVSvdCKUWgPoL0YCt621/Z73TM='} 2024-03-24T11:25:31.544 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-24T11:25:31.544 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:25:31.569 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-24T11:25:31.569 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:31.593 INFO:teuthology.orchestra.console.smithi031:Power off 2024-03-24T11:25:31.593 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:25:31.611 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-24T11:25:31.626 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-03-24T11:25:31.626 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:31.651 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-24T11:25:35.612 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:35.652 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:35.724 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-24T11:25:35.764 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-24T11:25:39.726 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:39.765 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:39.838 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-24T11:25:39.878 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-24T11:25:39.940 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-24T11:25:43.882 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:25:43.995 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-03-24T11:25:44.096 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-03-24T11:25:44.202 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3403.7498500347137 failure_reason: 'Command failed on smithi026 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e07c400a-e9c9-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c83b54c654284040abb8b56eb8888318 status: fail success: false 2024-03-24T11:25:44.203 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:25:44.307 INFO:teuthology.run:FAIL